Senior Data Engineer

ESPIRE INFOLABS (SINGAPORE) PTE. LTD.; Singapore; On-site; SGD 70,000 - 90,000; Full time

Job summary

A technology solutions company in Singapore is seeking a Data Engineer with at least 3 years of experience in data engineering. The role involves enhancing and implementing AWS-based data pipelines, monitoring their performance, and resolving any issues. Proficiency in Python, SQL, and familiarity with various AWS tools are essential. This position requires onsite work at MOHT premises when needed.

Qualifications

• At least 3 years hands-on working experience in data engineering.
• Experience in project execution with demonstrated technical expertise.
• Proficient in Python and SQL, with shell scripting knowledge.

Responsibilities

• Enhance existing data pipeline using industry-standard tools.
• Redesign and implement data pipeline to use AWS components.
• Implement monitoring to alert on pipeline issues.

Skills

Data engineering

Python

SQL

Shell scripting

ETL pipelines

AWS tools

Monitoring data pipeline

Airflow

Tools

AWS S3

AWS Glue

AWS EMR

AWS Lambda

AWS Step Functions

AWS Batch

Athena

NoSQL tools

AWS OpenSearch

Job description

Responsibilities

• Enhance the existing data pipeline using industry-standard tools and mechanisms.
• Redesign and implement data pipeline to use AWS managed components.
• Implement data pipeline monitoring to alert us if there is any issue with the pipeline.
• Investigate and resolve any issue with data pipeline.
• Document the data pipeline solution and how to operate the data pipeline.
• Implement unit testing and integration testing for data pipeline component.

Requirements

• At least 3 years hands-on working experience in data engineering, designing and building ETL pipelines.
• Experience in project execution and demonstrated technical expertise.
• Proficient in Python.
• SQL Knowledge.
• Shell scripts (Linux).
• Airflow or similar tool to create pipelines.
• AWS tools like S3, Glue, AWS EMR, Lambda, Step Functions.
• Familiar with tools like AWS Batch, Athena, file formats like Parquet, NoSQL knowledge and tools like AWS OpenSearch.
• The engineer should be able to physically work from MOHT premises whenever requested by the project team.