Big Data Engineer

Shirlyn Technology; Malaysia; On-site; MYR 100,000 - 150,000; Full time

Job summary

An established industry player is seeking a skilled Big Data Engineer II to join their dynamic team. This role is crucial for optimizing data infrastructure and tackling complex data challenges. You will build and deploy scalable data pipelines, ensuring data integrity, security, and quality across systems. With a focus on collaboration, you will work with global teams to enhance data solutions and drive business success. If you have a passion for data and a knack for problem-solving, this opportunity offers a chance to make a significant impact in a fast-paced environment.

Qualifications

• 5+ years of experience in building and optimizing data pipelines using big data technologies.
• Expertise in SQL and Python programming, focusing on maintainable code.

Responsibilities

• Collaborate with teams to understand data requirements and provide scalable solutions.
• Design and maintain efficient ETL pipelines for smooth data flow.

Skills

SQL

Python

ETL Pipelines

Data Quality Assurance

Big Data Technologies

Leadership

Scripting Languages

Unix/Linux

Education

Bachelor's Degree in Computer Science

Bachelor's Degree in Information Systems

Tools

Hive

Presto

Spark

Flink

Azure

AWS

GCP

Job description

Position Summary

We are looking for a full-time Big Data Engineer II to join our team at Shirlyn Technology in Petaling Jaya, Malaysia and Palo Alto, California. The Big Data team is at the heart of our operations, playing a critical role in scaling and optimizing our data infrastructure. As we continue to grow, we are seeking an experienced and driven Big Data Engineer to help us tackle complex data challenges, enhance data solutions, and ensure the security and quality of our data systems. You will be instrumental in building and deploying scalable data pipelines to ensure seamless data flow across systems, driving business success through reliable data solutions.

Job Responsibilities

• Collaborate with global teams in data, security, infrastructure, and business functions to understand data requirements and provide scalable solutions.
• Design, implement, and maintain efficient ETL (Extract, Transform, Load) pipelines to enable smooth data flow between systems.
• Conduct regular data quality checks to ensure accuracy, consistency, and completeness within our data pipelines.
• Continuously improve data pipeline performance and reliability, identifying and addressing any inefficiencies or bottlenecks.
• Ensure data integrity, security, privacy, and availability across all systems.
• Proactively monitor data pipelines, quickly diagnosing issues and taking action to resolve them and maintain system uptime.
• Conduct root cause analysis of data-related issues and work on long-term solutions to prevent recurring problems.
• Document pipeline designs, data processes, and troubleshooting procedures, keeping stakeholders informed with clear communication of updates.
• Provide on-call support for critical data operations, ensuring systems are running 24/7, with rotational responsibilities.

Job Requirements

• Bachelor’s degree in Computer Science, Information Systems, or a related technical field.
• 5+ years of hands-on experience building and optimizing data pipelines using big data technologies (Hive, Presto, Spark, Flink).
• Expertise in writing complex SQL queries and proficiency in Python programming, focusing on maintainable and clean code.
• Solid knowledge of scripting languages (e.g., Shell, Perl).
• Experience working in Unix/Linux environments.
• Familiarity with cloud platforms such as Azure, AWS, or GCP.
• High personal integrity and professionalism in handling confidential information, exercising sound judgment.
• Ability to remain calm and resolve issues swiftly during high-pressure situations, adhering to SLAs.
• Strong leadership skills, including the ability to guide junior team members and lead projects across the team.
• Excellent verbal and written communication skills, with the ability to collaborate effectively with remote teams globally.