Senior Data Engineer

IOTALENTS PTE. LTD.; Singapore; On-site; SGD 70,000 - 90,000; Full time

Job summary

A leading data solutions firm in Singapore is seeking an experienced professional to develop and optimize data pipelines in Hadoop. Key responsibilities include creating tested jobs using Control-M, preparing shell scripts for data archiving, and ensuring code optimization. The ideal candidate has a degree in Computer Science and a minimum of 5 years of experience in data pipeline development and Hadoop file formats. This role offers opportunities to work with advanced data technologies.

Qualifications

• 5+ years of experience in data pipeline development.
• Strong understanding of data models and entities.
• Experience optimizing code for performance.

Responsibilities

• Develop tested data pipelines to read/load data into Hadoop.
• Review and optimize code to resolve performance issues.
• Prepare shell scripts for data archiving in Hadoop.

Skills

Hadoop

PySpark

Shell scripting

Control-M

Education

Degree in Computer Science

Tools

Hadoop managed tables

Parquet

ORC

AVRO

Job description

Responsibilities

• Develop tested data pipelines to read the data from and load into Hadoop managed tables using yaml files.
• Review the code and also involve in code optimization to resolve performance issues.
• Develop, tested jobs using Control-M scheduler
• work on batch flow for production.
• Prepare shell scripts to do archive data in Hadoop environment.
• Worked on various Hadoop file formats like Parquet, ORC and AVRO files on HDFS using PySpark

Requirements

• Degree in Computer Science
• Have Good understanding of the data model and the entities used in the project, worked with data mappers to resolve mapping issues.
• Minimum 5 years of experience is needed