Data Engineer: Scalable Pipelines & Cloud Data

PERSOL SINGAPORE PTE. LTD.; Singapore; On-site; SGD 90,000 - 150,000; Full time

Job summary

PERSOL SINGAPORE PTE. LTD. is seeking a data engineer to design and implement data pipelines, collect and store data from multiple sources, and optimize SQL queries for reporting and analysis.

You will collaborate with data analysts/scientists to understand requirements, build scalable data stores on the cloud, and ensure data quality across large volumes of data. Experience with Hadoop ecosystem and CI/CD is preferred.

Qualifications

• Bachelor's degree in computer science, IT or related field.
• 4+ years of experience in data engineering.
• Strong SQL, ETL, Python and data extraction/manipulation skills.
• Experience with Kafka messaging tooling.
• Informatica development experience.
• Experience delivering complex ETL and data extraction projects from scratch.
• Fluent in Python, Spark, HiveQL, Hadoop ecosystem; Unix/Linux/HDFS environment proficiency; SQL and RDBMS.
• Good working knowledge of Git and CI/CD pipelines.

Responsibilities

• Assist design and implementation of data pipelines to collect, process, and store data from various sources.
• Write and optimize SQL queries for reporting and analysis.
• Collaborate with data analysts/scientists to understand data requirements and provide data support.
• Organize, build and maintain data stores of structured and unstructured data on a Cloud environment.
• Build data pipelines to enable scalable processing of large volumes of data.
• Develop scalable, high-performance application architectures using modern frameworks.
• Work with stakeholders to gather requirements and translate them into solutions; monitor and troubleshoot pipelines to ensure data quality.
• Enhance and maintain existing applications; identify areas for optimization.
• Document data processes, workflows and models; maintain deployment documentation.

Skills

Data engineering

Data pipelines

SQL

Python

ETL

Spark

Hadoop ecosystem

Unix/Linux

RDBMS

Education

Bachelor's degree in computer science/IT or related field

Tools

Kafka

Informatica

Git

CI/CD

Teradata

HDFS

Job description

PERSOL SINGAPORE PTE. LTD. is seeking a data engineer to design and implement data pipelines, collect and store data from multiple sources, and optimize SQL queries for reporting and analysis.

You will collaborate with data analysts/scientists to understand requirements, build scalable data stores on the cloud, and ensure data quality across large volumes of data. Experience with Hadoop ecosystem and CI/CD is preferred.