Senior Data Engineer: Scalable Spark & Cloud Data Pipelines

UNISYNC SYSTEMS PTE. LTD.; Singapore; On-site; SGD 120,000 - 180,000; Full time

Job summary

UNISYNC SYSTEMS PTE. LTD. is seeking a Senior Data Engineer to design, develop and maintain scalable data pipelines for large-volume datasets in a cloud-first environment.

You will work with Spark, PySpark, Scala and Python to build ETL/ELT processes and optimize performance and cost. Collaborate with stakeholders to translate requirements into robust data models, storage architectures and data lakes, using Delta Lake or Snowflake, while implementing CI/CD and testing with Airflow, Jenkins and

Qualifications

• 6+ years of experience in Data Engineering / Big Data.
• Strong hands-on experience with Apache Spark / PySpark.
• Strong programming skills in Python and/or Scala.
• Experience with Hadoop, HDFS and Hive.
• Experience with cloud data platforms (Azure Databricks, Azure Data Factory, AWS EMR).
• Experience with Kafka / real-time streaming is a plus.
• Hands-on experience with Airflow and data pipeline orchestration.
• Knowledge of Snowflake, Teradata or similar enterprise databases.

Responsibilities

• Design, develop and maintain scalable data pipelines for large-volume datasets.
• Develop data transformation and processing apps using Spark, PySpark, Scala and Python.
• Build and optimize data pipelines using cloud services like Azure Databricks and Azure Data Factory.
• Work with Hadoop, HDFS, Hive, Snowflake, Teradata and Data Lake environments.
• Develop batch and real-time data processing with Spark Structured Streaming and Kafka.
• Perform ETL/ELT across heterogeneous source and target systems.
• Develop and optimize Spark SQL, HiveQL and SQL queries for performance and cost.
• Design data models, partitioning strategies and scalable storage architectures.
• Build and manage workflow orchestration with Apache Airflow.
• Implement CI/CD pipelines and automated testing using Jenkins, Docker, GitHub Actions and pytest.
• Troubleshoot data pipelines, performance and production issues and implement durable solutions.
• Collaborate with stakeholders to deliver data engineering solutions.
• Ensure data quality, reliability, security and operational stability.

Skills

Spark / PySpark

Python

Scala

SQL

Airflow

CI/CD

Docker

Cloud platforms

Job description

UNISYNC SYSTEMS PTE. LTD. is seeking a Senior Data Engineer to design, develop and maintain scalable data pipelines for large-volume datasets in a cloud-first environment.

You will work with Spark, PySpark, Scala and Python to build ETL/ELT processes and optimize performance and cost. Collaborate with stakeholders to translate requirements into robust data models, storage architectures and data lakes, using Delta Lake or Snowflake, while implementing CI/CD and testing with Airflow, Jenkins and