Spark-Powered Data Engineer: ETL & PySpark Pipelines

Hong Kong Unison Limited; Singapore; On-site; SGD 80,000 - 120,000; Full time

Job summary

Unison Consulting is seeking a skilled Data Engineer to design, develop, and optimize scalable data pipelines and ETL processes in a Singapore-based role. The candidate will work with Python, PySpark, SQL, and cloud data platforms to handle large-scale datasets and ensure data quality and performance.

Responsibilities include building data models and warehouses, optimizing Spark jobs, and collaborating with analysts and business teams. Strong scripting, CI/CD, and cloud experience are preferred.

Qualifications

• Proficient in Python programming with hands-on PySpark experience.
• Strong SQL skills with query optimization and data transformations.
• Experience building ETL/ELT pipelines and data processing solutions.
• Solid understanding of data warehousing concepts and relational databases.
• Knowledge of Linux/Unix environments and shell scripting.
• Experience with Git and CI/CD processes.
• Analytical mindset with strong problem-solving skills.

Responsibilities

• Design, develop, and maintain scalable ETL/ELT data pipelines using Python and PySpark.
• Develop high-performance data processing solutions for structured and unstructured data.
• Write optimized SQL queries, stored procedures, and data transformations.
• Build and maintain data models, data marts, and data warehouses.
• Perform data cleansing, validation, and quality checks.
• Optimize Spark jobs for performance and scalability.
• Integrate data from APIs, databases, and cloud storage.
• Collaborate with analysts, data scientists, and business teams to deliver data solutions.
• Troubleshoot production issues and perform root cause analysis.
• Ensure data governance, security, and best engineering practices.
• Participate in code reviews and maintain technical documentation.

Skills

Python

PySpark

SQL

ETL/ELT

Data warehousing concepts

Relational databases

Linux/Unix scripting

Git & CI/CD

Analytical & problem solving

Education

Bachelor's or Master's in CS/IT/Engineering

Tools

Airflow

Databricks

Hadoop/Hive ecosystem

Cloud platforms (AWS/Azure/GCP)

Docker & Kubernetes

Job description

Unison Consulting is seeking a skilled Data Engineer to design, develop, and optimize scalable data pipelines and ETL processes in a Singapore-based role. The candidate will work with Python, PySpark, SQL, and cloud data platforms to handle large-scale datasets and ensure data quality and performance.

Responsibilities include building data models and warehouses, optimizing Spark jobs, and collaborating with analysts and business teams. Strong scripting, CI/CD, and cloud experience are preferred.