Data Engineer

COMBUILDER PTE LTD; Singapore; On-site; SGD 70,000 - 100,000; Full time

Job summary

A leading data engineering firm in Singapore is seeking a skilled Data Engineer to design and implement enterprise-scale data architectures. This role involves developing and maintaining ETL/ELT pipelines, providing technical guidance to junior engineers, and optimizing pipeline performance. The ideal candidate has 5-7 years of experience in data engineering, proficiency in AWS and Databricks, and a strong background in data manipulation and validation processes.

Qualifications

• 5-7 years of data engineering experience with AWS-native implementations.
• Proven experience working on ETL migration projects.
• Hands-on experience implementing Databricks notebooks using PySpark, Python, and SQL.

Responsibilities

• Design and implement enterprise-scale data architectures.
• Develop and maintain ETL/ELT pipelines for large volumes of data.
• Monitor and optimize data pipeline performance.

Skills

AWS cloud platform expertise

Databricks platform proficiency

Python

SQL

ETL/ELT migration project experience

Data warehouse design

Real-time streaming technologies

Data exchange platforms

Education

Bachelor's Degree in Computer Science, Engineering, Data Science, or related field

Master's Degree in Data Engineering, Computer Science, or MBA

Tools

AWS (RedShift, S3, Glue, EMR, Kinesis, Lambda)

Databricks

Job description

Responsibilities

• Date Architecture & Engineering: Design and implement enterprise-scale date architectures, including data lakes, warehouses, and real-time streaming platforms. Develop and maintain ETL/ELT pipelines that efficiently process large volumes of structured and unstructured date from diverse sources. Ensure date quality, governance, and security standards are embedded throughout all the data engineering processes.
• Technical Implementation: Hands-on development of Databricks notebooks using PySpark, Python, and SQL for ETL automation. Create and optimize PySpark scripts for efficient data extraction, transformation, and loading from large datasets. Implement custom data manipulation, validation, and error handling solutions to enhance ETL robustness.
• Technical Guidance: Provide technical mentorship to junior date engineers and analysts. Lead code review, establish best practices, and drive adoption of modern data engineering tools and methodologies. Collaborate with cross-function teams including data scientists, analysts, and software engineers to deliver integrated solutions.
• Performance & Optimisation: Monitor and optimise data pipeline and performance, implementing solutions for scalability and cost-effectiveness. Conduct testing, debugging, and troubleshooting of data transformation processes. Verify data integrity throughout pipeline stages and resolve complex technical issues.

Requirements

• Bachelor's Degree in Computer Science, Engineering, Data Science, or related field.
• Master's Degree in Data Engineering, Computer Science, or MBA (preferred)
• Relevant diploma or certificate in AWS cloud technologies or data engineering.
• 5-7 years of data engineering experience with AWS-native implementations.
• Proven experience working as part of a date engineering team on ETL migration projects.
• Hand-on experience implementing Databricks notebooks using PySpark, Python, and SQL for ETL automation.
• Demonstrated expertise developing PySpark scripts for efficient data extraction, transformation, and loading from large datasets.
• Experience utilising Python for custom data manipulation, validation, and error handling ETL processes.
• Proficiency employing SQL for complex joins, aggregations, and database operations within Databricks environments.
• Track record of testing, debugging, and optimising data transformation processes for accuracy and performance.
• Experience verifying data integrity throughout pipeline stages and resolving troubleshooting issues.
• Proven ability collaborating with cross-functional teams to align ETL migrations tasks with project goals and deadlines.
• Experience in government sector data projects and compliance requirements (preferred).
• Experience in HR domain including workforce analytics, payroll systems, and employee data management (preferred)
• Experience providing technical mentorship to junior team members (preferred)
• Technical Skills:
• AWS cloud platform expertise (RedShift, S3, Glue, EMR, Kinesis, Lambda)
• Databricks platform proficiency with PySpark, Python, and SQL implementation.
• Data warehouse design and implementation (Amazon RedShift, Snowflake on AWS)
• ETL/ELT migration project experience and pipeline development.
• Real-time streaming technologies (Kinesis Data Streams, Kafka)
• Data exchange platforms and API integration (AWS Data Exchange, Partner APIs)