Big Data Engineer

Position Summary

We are looking for a full-time Big Data Engineer II to join our team at Shirlyn Technology in Petaling Jaya, Malaysia and Palo Alto, California. The Big Data team is at the heart of our operations, playing a critical role in scaling and optimizing our data infrastructure. As we continue to grow, we are seeking an experienced and driven Big Data Engineer to help us tackle complex data challenges, enhance data solutions, and ensure the security and quality of our data systems. You will be instrumental in building and deploying scalable data pipelines to ensure seamless data flow across systems, driving business success through reliable data solutions.

Job Responsibilities

• Collaborate with global teams in data, security, infrastructure, and business functions to understand data requirements and provide scalable solutions.
• Design, implement, and maintain efficient ETL (Extract, Transform, Load) pipelines to enable smooth data flow between systems.
• Conduct regular data quality checks to ensure accuracy, consistency, and completeness within our data pipelines.
• Continuously improve data pipeline performance and reliability, identifying and addressing any inefficiencies or bottlenecks.
• Ensure data integrity, security, privacy, and availability across all systems.
• Proactively monitor data pipelines, quickly diagnosing issues and taking action to resolve them and maintain system uptime.
• Conduct root cause analysis of data-related issues and work on long-term solutions to prevent recurring problems.
• Document pipeline designs, data processes, and troubleshooting procedures, keeping stakeholders informed with clear communication of updates.
• Provide on-call support for critical data operations, ensuring systems are running 24/7, with rotational responsibilities.

Job Requirements

• Bachelor’s degree in Computer Science, Information Systems, or a related technical field.
• 5+ years of hands-on experience building and optimizing data pipelines using big data technologies (Hive, Presto, Spark, Flink).
• Expertise in writing complex SQL queries and proficiency in Python programming, focusing on maintainable and clean code.
• Solid knowledge of scripting languages (e.g., Shell, Perl).
• Experience working in Unix/Linux environments.
• Familiarity with cloud platforms such as Azure, AWS, or GCP.
• High personal integrity and professionalism in handling confidential information, exercising sound judgment.
• Ability to remain calm and resolve issues swiftly during high-pressure situations, adhering to SLAs.
• Strong leadership skills, including the ability to guide junior team members and lead projects across the team.
• Excellent verbal and written communication skills, with the ability to collaborate effectively with remote teams globally.