Data Engineer

ELPA ASIA PTE. LTD.; Singapore; On-site; SGD 70,000 - 100,000; Full time

Job summary

A leading data solutions company in Singapore seeks a Data Engineer to enhance and implement data pipelines using AWS managed components. Ideal candidates have over 5 years of experience in data engineering, proficient in Python, SQL, and AWS tools. Responsibilities include monitoring data pipelines, troubleshooting issues, and implementing testing. Join a dynamic team committed to innovative data solutions.

Qualifications

• 5+ years of hands-on experience in data engineering.
• Experience in project execution and technical expertise.
• Proficient in Python, SQL, and shell scripts.

Responsibilities

• Enhance the existing data pipeline using industry-standard tools.
• Redesign data pipeline using AWS managed components.
• Implement monitoring for the data pipeline.
• Investigate and resolve data pipeline issues.
• Document data pipeline solutions.
• Implement unit and integration testing.

Skills

Data engineering

ETL pipeline design

Python

SQL

Scripting (Linux)

AWS (S3, Glue, EMR, Lambda)

Tools

Airflow

AWS Batch

AWS Athena

AWS OpenSearch

Job description

Responsibilities

• Enhance the existing data pipeline using industry-standard tools and mechanisms.
• Redesign and implement data pipeline to use AWS managed components.
• Implement data pipeline monitoring to alert us if there is any issue with the pipeline.
• Investigate and resolve any issue with data pipeline.
• Document the data pipeline solution and how to operate the data pipeline.
• Implement unit testing and integration testing for data pipeline component.

Requirements

• At least 5 years hands-on working experience in data engineering, designing and building ETL pipelines.
• Experience in project execution and demonstrated technical expertise.
• Proficient in Python, SQL and Schell scripts (Linux).
• Airflow or similar tool to create pipelines.
• AWS tools like S3, Glue, AWS EMR, Lambda, Step functions.
• Familiar with tools like AWS Batch, Athena, file formats like Parquet, NoSQL knowledge and tools like AWS OpenSearch.