Senior Data Engineer: Spark & Cloud Data Pipelines

UNISONEDGE CONSULTING PTE. LTD.; Singapore; On-site; SGD 120,000 - 180,000; Full time

Job summary

UNISONEDGE CONSULTING PTE. LTD. is seeking an experienced Data Engineer to design and maintain scalable data pipelines for large-volume datasets.

You will develop transformations with Spark, PySpark, Scala and Python while optimizing pipelines on Azure Databricks, Data Factory, and AWS EMR across diverse cloud environments. You will work with Hadoop, HDFS, Hive, Snowflake and Data Lake technologies, delivering batch and real-time processing using Spark Streaming and Kafka.

Qualifications

• 6+ years of experience in Data Engineering / Big Data Engineering.
• Strong hands-on experience with Apache Spark / PySpark.
• Strong programming skills in Python and/or Scala.
• Good experience with Hadoop, HDFS and Hive.
• Experience developing ETL/ELT and data ingestion pipelines.
• Strong SQL and data processing skills.
• Experience with Azure Databricks, Azure Data Factory, AWS EMR or equivalent cloud platforms.
• Experience with Kafka / real-time streaming is an advantage.
• Hands-on experience with Airflow and data pipeline orchestration.
• Experience with Snowflake, Teradata, SQL Server or other enterprise databases.
• Good understanding of Data Lake, Delta Lake, Data Warehousing and Data Modelling.
• Experience with Git, CI/CD, Docker and automated testing.

Responsibilities

• Design, develop and maintain scalable data pipelines and ingestion frameworks for large-volume datasets.
• Develop data transformation apps using Spark, PySpark, Scala and Python.
• Build and optimize pipelines using Azure Databricks, Data Factory, AWS EMR and related cloud services.
• Work with Hadoop, HDFS, Hive, Snowflake, Teradata and Data Lake environments.
• Develop batch and real-time data processing using Spark Structured Streaming and Kafka.
• Perform ETL/ELT across heterogeneous source and target systems.
• Develop and optimize Spark SQL, HiveQL and SQL queries for performance and cost efficiency.
• Design data models, partitioning strategies and scalable storage architectures.
• Build and manage workflow orchestration using Apache Airflow.
• Implement CI/CD pipelines and automated testing using Jenkins, Docker, GitHub Actions and pytest.
• Troubleshoot data pipeline, performance and production issues and implement sustainable solutions.
• Collaborate with business stakeholders, architects and technology teams to understand requirements and deliver data engineering solutions.
• Ensure data quality, reliability, security and operational stability across enterprise data platforms.

Skills

Spark / PySpark

Python / Scala

Hadoop / HDFS / Hive

ETL / ELT pipelines

SQL / data processing

Azure Databricks / Data Factory

AWS EMR

Kafka / real-time streaming

Airflow

Snowflake / Teradata / SQL Server

Delta Lake / Data Lake concepts

Git / CI/CD / Docker

Tools

Airflow

Docker

GitHub Actions

pytest

Jenkins

Job description

UNISONEDGE CONSULTING PTE. LTD. is seeking an experienced Data Engineer to design and maintain scalable data pipelines for large-volume datasets.

You will develop transformations with Spark, PySpark, Scala and Python while optimizing pipelines on Azure Databricks, Data Factory, and AWS EMR across diverse cloud environments. You will work with Hadoop, HDFS, Hive, Snowflake and Data Lake technologies, delivering batch and real-time processing using Spark Streaming and Kafka.