Senior Data Engineer — Spark, Cloud Pipelines & Real-Time

REGTECH INSIGHT PTE. LTD.; Singapore; On-site; SGD 120,000 - 180,000; Full time

Job summary

REGTECH INSIGHT PTE. LTD. is seeking an experienced Data Engineer to design, build and optimize scalable data pipelines for large datasets. You will develop batch and real-time processing using Spark, PySpark, Scala and Python, and leverage Azure Databricks, ADF and AWS EMR across diverse data platforms.

You will work with Hadoop ecosystems, Snowflake and data lake environments, crafting efficient SQL queries and robust data models while implementing Airflow-driven workflows and CI/CD practices.

Qualifications

• 6+ years of data engineering / big data experience.
• Hands-on with Spark / PySpark.
• Strong Python and/or Scala programming.
• Experience with Hadoop, HDFS and Hive.
• ETL/ELT data pipelines development.
• Strong SQL and data processing skills.
• Experience with Azure Databricks, ADF, AWS EMR.
• Kafka / real-time streaming is a plus.
• Airflow for workflow orchestration.
• Experience with Snowflake, Teradata, SQL Server or similar.
• Knowledge of Data Lake, Delta Lake, DW and data modelling.
• Git, CI/CD, Docker and automated testing.

Responsibilities

• Design, develop and maintain scalable data pipelines and ingestion frameworks for large-volume datasets.
• Develop data transformation and processing applications using Apache Spark, PySpark, Scala and Python.
• Build and optimize pipelines using AzureDatabricks, Azure Data Factory, AWS EMR and cloud services.
• Work with Hadoop, HDFS, Hive, Snowflake, Teradata and Data Lake environments.
• Develop batch and real-time data processing using Spark Structured Streaming and Kafka.
• Perform ETL/ELT and data ingestion across heterogeneous sources and targets.
• Develop and optimize Spark SQL, HiveQL and SQL queries for performance and cost efficiency.
• Design data models, partitioning strategies and scalable storage architectures.
• Build and manage workflow orchestration using Apache Airflow.
• Implement CI/CD pipelines and automated testing with Jenkins, Docker, GitHub Actions and pytest.
• Troubleshoot data pipeline, performance and production issues; implement sustainable solutions.
• Collaborate with stakeholders, architects and tech teams to deliver data engineering solutions.
• Ensure data quality, reliability, security and operational stability across platforms.

Skills

Data engineering

Big data engineering

Spark PySpark

Python/Scala

Hadoop/HDFS/Hive

ETL/ELT pipelines

SQL

Cloud data platforms

Kafka streaming

Airflow

Snowflake/Teradata/SQL Server

Data Lake/Delta Lake

Git CI/CD Docker

Tools

Azure Databricks

Azure Data Factory

AWS EMR

Job description

REGTECH INSIGHT PTE. LTD. is seeking an experienced Data Engineer to design, build and optimize scalable data pipelines for large datasets. You will develop batch and real-time processing using Spark, PySpark, Scala and Python, and leverage Azure Databricks, ADF and AWS EMR across diverse data platforms.

You will work with Hadoop ecosystems, Snowflake and data lake environments, crafting efficient SQL queries and robust data models while implementing Airflow-driven workflows and CI/CD practices.