Senior Data Engineer

SEDHA CONSULTING PTE. LTD.; Singapore; On-site; SGD 60,000 - 90,000; Full time

Job summary

A data consulting firm in Singapore is seeking an experienced Data Engineer to enhance and redesign existing data pipelines using AWS components. The ideal candidate should have a minimum of 3 years' experience in data engineering, proficiency in Python, and a strong understanding of ETL processes. This role requires physical presence at project sites when requested.

Qualifications

• At least 3 years hands-on working experience in data engineering, designing and building ETL pipelines.
• Experience in project execution and demonstrated technical expertise.
• Familiar with tools like AWS Batch and Athena, and file formats like Parquet.

Responsibilities

• Enhance the existing data pipeline using industry-standard tools and mechanisms.
• Redesign and implement data pipeline to use AWS managed components.
• Implement data pipeline monitoring to alert us if there is any issue with the pipeline.

Skills

Data engineering

ETL pipelines

Python

SQL

Shell scripting

AWS tools

Tools

Airflow

AWS Glue

AWS EMR

AWS Lambda

Job description

Responsibilities

• Enhance the existing data pipeline using industry-standard tools and mechanisms.
• Redesign and implement data pipeline to use AWS managed components.
• Implement data pipeline monitoring to alert us if there is any issue with the pipeline.
• Investigate and resolve any issue with data pipeline.
• Document the data pipeline solution and how to operate the data pipeline.
• Implement unit testing and integration testing for data pipeline component.

Requirements

• At least 3 years hands-on working experience in data engineering, designing and building ETL pipelines.
• Experience in project execution and demonstrated technical expertise.
• Proficient in Python.
• SQL Knowledge.
• Shell scripts (Linux).
• Airflow or similar tool to create pipelines.
• AWS tools like S3, Glue, AWS EMR, Lambda, Step functions.
• Familiar with tools like AWS Batch, Athena, file formats like Parquet, NoSQL knowledge and tools like AWS OpenSearch.
• The engineer should be able to physically work from MOHT premises whenever requested by the project team.