Software Engineer

Combuilder Pte Ltd; Singapore; On-site; SGD 80,000 - 120,000; Full time

Job summary

A leading data engineering company in Singapore is seeking an experienced Data Engineer. You will provide L3 support for Apache Spark and Airflow, design efficient data pipelines, and optimize data infrastructure using AWS technologies. Ideal candidates have over 10 years of experience in data engineering, strong skills in Java, Python, and a deep understanding of AWS. The role demands excellent communication skills and the ability to thrive in a collaborative environment.

Qualifications

• 10+ years of experience in software development, ETL, and data engineering.
• Strong knowledge of AWS services: S3, Glue, EMR, Redshift.
• Excellent problem-solving skills and collaborative in nature.

Responsibilities

• Provide L3 support for Apache Spark and Airflow related issues.
• Design and maintain complex workflow and data pipelines.
• Troubleshoot data pipelines to ensure smooth operation.

Skills

Apache Spark

Apache Airflow

Java

Python

PySpark

AWS (S3, Glue, EMR, Redshift)

SQL

ETL processes

Education

Bachelor’s degree in Computer Science, Engineering, or a related field

Job description

Responsibilities:

• Provide L3 support for all Apache Spark and Airflow related issues for the Data Platform.
• Design, develop, and maintain complex workflow, data pipelines, processes, and system architecture using Java, Python, PySpark, Apache Spark, Airflow and AWS technologies for efficient data processing and orchestration.
• Work closely with AWS services like S3, Glue, EMR, Redshift, and other related technologies to design and optimize data infrastructure.
• Implement data integration and transformation processes to ensure optimal performance and reliability of data pipelines.
• Optimize and fine-tune existing data pipelines / Airflow to improve efficiency, scalability, and maintainability.
• Troubleshoot and resolve issues related to data pipelines, ensuring smooth operation and minimal downtime.
• Develop and maintain documentation for data pipelines, processes, and system architecture.
• Stay updated with the latest industry trends and best practices related to data engineering and AWS services.

Requirements:

• Bachelor’s degree in Computer Science, Engineering, or a related field.
• At least 10 years of experience in software development, ETL, and data engineering, specifically working with Apache Spark, Apache Airflow and AWS technologies i.e. strong knowledge of AWS services, particularly S3, Glue, EMR, Redshift, and AWS Lambda.
• Proficient in the design and support of complex workflow for various business divisions to load data marts tables from S3 staging layer to AWS Redshift on daily basis in a 24 x 7 BAU environment.
• Strong proficiency (at least 8 years) in Java, Python, PySpark and SQL for data processing and manipulation, with experience with optimizing and scaling data pipelines for performance and efficiency.
• Strong understanding of Snowflake Data Lake is preferred.
• Strong understanding of data modelling, ETL processes and tools, and data warehousing concepts and tools.
• Excellent problem-solving skills and ability to work in a fast-paced, collaborative environment.
• Effective communication skills and the ability to articulate technical concepts to non-technical stakeholders.