Senior Data Engineer: Spark, Cloud Pipelines & Real-Time Data

Riskdata Consulting; Singapore; On-site; SGD 120,000 - 180,000; Full time

Job summary

Riskdata Consulting is seeking a seasoned Data Engineer to design, build and optimize scalable data pipelines for large datasets. You will work with Spark, PySpark, Scala and Python to develop batch and streaming solutions, leveraging Azure Databricks, Data Factory and AWS EMR across Hadoop/Hive/Snowflake environments.

The role emphasizes data modeling, performance-focused SQL, and robust data governance with Airflow orchestration, CI/CD pipelines, and automated testing.

Qualifications

• 6+ years of experience in Data Engineering / Big Data Engineering.
• Strong hands-on experience with Apache Spark / PySpark.
• Strong programming skills in Python and/or Scala.
• Good experience with Hadoop, HDFS and Hive.
• Experience developing ETL/ELT and data ingestion pipelines.
• Strong SQL and data processing skills.
• Experience with Azure Databricks, Azure Data Factory, AWS EMR or equivalent cloud data platforms.
• Experience with Kafka / real-time streaming is an advantage.
• Hands-on experience with Airflow and data pipeline orchestration.
• Experience with Snowflake, Teradata, SQL Server or other enterprise databases.
• Good understanding of Data Lake, Delta Lake, Data Warehousing and Data Modelling.
• Experience with Git, CI/CD, Docker and automated testing.

Responsibilities

• Design, develop and maintain scalable data pipelines and ingestion frameworks for large datasets.
• Develop data transformation apps using Spark, PySpark, Scala and Python.
• Build and optimize pipelines on Azure Databricks, Data Factory, EMR.
• Work with Hadoop, HDFS, Hive, Snowflake, Teradata and Data Lake.
• Develop batch and real-time processing with Spark Structured Streaming and Kafka.
• Perform ETL/ELT and ingestion across heterogeneous sources.
• Develop and optimize Spark SQL, HiveQL and SQL queries for performance and cost.
• Design data models, partitioning strategies and scalable storage.
• Manage workflow orchestration with Apache Airflow.
• CI/CD and automated testing with Jenkins, Docker, GitHub Actions and pytest.
• Troubleshoot and resolve data pipeline issues and optimize performance.
• Collaborate with stakeholders to deliver data solutions.
• Ensure data quality, reliability, security and stability across enterprise data platforms.

Skills

Data Engineering

Spark / PySpark

Python/Scala

Hadoop/HDFS/Hive

ETL/ELT

SQL

Cloud platforms

Kafka

Airflow

Databases

Data Modeling

Git/CI-CD/Docker

Tools

Apache Airflow

Docker

Git

CI/CD

Job description

Riskdata Consulting is seeking a seasoned Data Engineer to design, build and optimize scalable data pipelines for large datasets. You will work with Spark, PySpark, Scala and Python to develop batch and streaming solutions, leveraging Azure Databricks, Data Factory and AWS EMR across Hadoop/Hive/Snowflake environments.

The role emphasizes data modeling, performance-focused SQL, and robust data governance with Airflow orchestration, CI/CD pipelines, and automated testing.