Data Engineer

PERSOL SINGAPORE PTE. LTD.; Singapore; On-site; SGD 90,000 - 150,000; Full time

Job summary

PERSOL SINGAPORE PTE. LTD. is seeking a data engineer to design and implement data pipelines, collect and store data from multiple sources, and optimize SQL queries for reporting and analysis.

You will collaborate with data analysts/scientists to understand requirements, build scalable data stores on the cloud, and ensure data quality across large volumes of data. Experience with Hadoop ecosystem and CI/CD is preferred.

Qualifications

• Bachelor's degree in computer science, IT or related field.
• 4+ years of experience in data engineering.
• Strong SQL, ETL, Python and data extraction/manipulation skills.
• Experience with Kafka messaging tooling.
• Informatica development experience.
• Experience delivering complex ETL and data extraction projects from scratch.
• Fluent in Python, Spark, HiveQL, Hadoop ecosystem; Unix/Linux/HDFS environment proficiency; SQL and RDBMS.
• Good working knowledge of Git and CI/CD pipelines.

Responsibilities

• Assist design and implementation of data pipelines to collect, process, and store data from various sources.
• Write and optimize SQL queries for reporting and analysis.
• Collaborate with data analysts/scientists to understand data requirements and provide data support.
• Organize, build and maintain data stores of structured and unstructured data on a Cloud environment.
• Build data pipelines to enable scalable processing of large volumes of data.
• Develop scalable, high-performance application architectures using modern frameworks.
• Work with stakeholders to gather requirements and translate them into solutions; monitor and troubleshoot pipelines to ensure data quality.
• Enhance and maintain existing applications; identify areas for optimization.
• Document data processes, workflows and models; maintain deployment documentation.

Skills

Data engineering

Data pipelines

SQL

Python

ETL

Spark

Hadoop ecosystem

Unix/Linux

RDBMS

Education

Bachelor's degree in computer science/IT or related field

Tools

Kafka

Informatica

Git

CI/CD

Teradata

HDFS

Job description

Responsibilities

• Assist in the design and implementation of data pipelines to collect, process, and store data from various sources.
• Write and optimize SQL queries to extract and manipulate data for reporting and analysis.
• Collaborate with data analysts and data scientists to understand data requirements and provide the necessary data support.
• Organize, build and maintain data stores of structured and unstructured data on a Cloud environment.
• Build data pipelines to enable scalable processing of large volumes of data.
• Develop scalable and high-performance application architectures using modern development frameworks.
• Work closely with business stakeholders and project leads to gather requirements and convert them into effective solutions. Monitor and troubleshoot data pipelines and workflows to ensure data quality and reliability
• Enhance and maintain existing applications, identifying areas for optimization and improvement.
• Document data processes, workflows, and data models to maintain clear communication within the team
• Maintain technical documentation related to application design, workflows, and deployment.

Skills/Requirement

• Bachelor's degree in computer science, Information Technology, or a related field.
• 4+ years of experience in data engineering
• Strong knowledge in SQL, ETL Teradata, Python and experience with writing queries for data extraction and manipulation.
• knowledge of managing messaging tools like Kafka.
• Experience in informatica development.
• Experience implementing complex ETL and data extraction projects from scratch.
• Fluency in Python, Spark, HIVE-QL, Hadoop eco-system, data-centric workflow management tools Unix/Linux/HDFS environments Proficiency in SQL and RDBMS
• Good working knowledge of Git and CI/CD pipelines

We regret to inform you that only shortlisted candidates will be contacted.