Data Engineer

CLOUD KINETICS CONSULTING PTE. LTD.; Singapore; On-site; SGD 60,000 - 80,000; Full time

Job summary

A leading data solutions firm located in Singapore is looking for a Data Engineer to develop and maintain ETL/ELT pipelines for processing both structured and unstructured data. The ideal candidate will have a minimum of 1-2 years of data engineering experience with Azure-native implementations and demonstrated skills in building pipelines using ADF, Synapse, and Delta Lake. Expertise in PySpark and SQL is essential for ensuring data quality and performance optimization.

Qualifications

• Minimum 1-2 years of data engineering experience with Azure-native implementations.
• Proven experience working as part of a data engineering team.
• Hands-on experience building pipelines using ADF, Synapse, and Delta Lake.
• Demonstrated expertise developing PySpark scripts for data ETL processes.
• Experience utilizing Python for data manipulation and validation in ETL processes.
• Proficiency employing SQL for complex database operations within Databricks.

Responsibilities

• Develop and maintain ETL/ELT pipelines for structured and unstructured data.
• Ensure data quality, governance, and security in data engineering processes.
• Hands-on development of notebooks for ETL automation using PySpark and SQL.
• Monitor and optimize data pipeline performance for cost-effectiveness.
• Conduct testing, debugging, and troubleshooting of data processes.

Skills

Data engineering experience

PySpark

Python

SQL

Azure-native implementations

ADF

Synapse

Delta Lake

Job description

Data Architecture & EngineeringDevelop and maintain ETL/ELT pipelines that efficiently process large volumes of structured and unstructured data from diverse sources.

Ensure data quality, governance, and security standards are embedded throughout all data engineering processes.

Technical Implementation

Hands-on development of notebooks using PySpark, Python, and SQL for ETL automation.

Create and optimize PySpark scripts for efficient data extraction, transformation, and loading from large datasets.

Implement custom data manipulation, validation, and error handling solutions to enhance ETL robustness.

Performance & Optimization

Monitor and optimize data pipeline performance, implementing solutions for scalability and cost-effectiveness.

Conduct testing, debugging, and troubleshooting of data transformation processes.

Verify data integrity throughout pipeline stages and resolve complex technical issues.

Requirements

Minimum 1-2 years of data engineering experience with Azure-native implementations

Proven experience working as part of a data engineering team.

Hands on experience building pipeline using ADF, Synapse and Delta lake.

Demonstrated expertise developing PySpark scripts for efficient data extraction, transformation, and loading from large datasets

Experience utilizing Python for custom data manipulation, validation, and error handling in ETL processes

Proficiency employing SQL for complex joins, aggregations, and database operations within Databricks environments

Track record of testing, debugging, and optimizing data transformation processes for accuracy and performance

Experience verifying data integrity throughout pipeline stages and resolving troubleshooting issues