Senior Data Engineer - Real-Time Pipelines & Spark

UARROW PTE. LTD.; Singapore; On-site; SGD 110,000 - 190,000; Full time

Job summary

UARROW PTE. LTD. is seeking an experienced Data Engineer to design, build and optimize scalable data pipelines for large datasets. You will work across Spark/PySpark, Python/Scala, and cloud services to deliver reliable ingestion, transformation and warehousing capabilities.

You will deploy batch and real-time processing using Spark Structured Streaming and Kafka, collaborate with cross‑functional teams, and ensure data quality, security and cost efficiency in a fast-paced enterprise environment.

Qualifications

• 6+ years of experience in Data Engineering / Big Data Eng.
• Strong hands‑on with Apache Spark / PySpark.
• Strong programming in Python and/or Scala.
• Good experience with Hadoop, HDFS and Hive.
• Experience developing ETL/ELT and data ingestion pipelines.
• Strong SQL and data processing skills.
• Experience with Azure Databricks, Azure Data Factory, AWS EMR.
• Experience with Kafka / real‑time streaming is an advantage.
• Hands‑on experience with Airflow and data pipeline orchestration.
• Experience with Snowflake, Teradata, SQL Server or other enterprise databases.

Responsibilities

• Design, develop and maintain scalable data pipelines and ingestion frameworks for large-volume datasets.
• Develop data transformation and processing applications using Spark, PySpark, Scala and Python.
• Build and optimize pipelines using Azure Databricks, Azure Data Factory, AWS EMR and related clouds.
• Work with Hadoop, HDFS, Hive, Snowflake, Teradata and Data Lake environments.
• Develop batch and real-time data processing using Spark Structured Streaming and Kafka.
• Perform data extraction, transformation and loading across heterogeneous sources.
• Develop and optimize Spark SQL, HiveQL and SQL queries for performance and cost efficiency.
• Design data models, partitioning strategies and scalable storage architectures.
• Build and manage workflow orchestration using Apache Airflow.
• Implement CI/CD and automated testing using Jenkins, Docker, GitHub Actions and pytest.
• Troubleshoot data pipelines and production issues; deliver sustainable solutions.
• Collaborate with stakeholders to understand requirements and deliver data engineering solutions.
• Ensure data quality, reliability, security and operational stability across platforms.

Skills

Data Engineering / Big Data Eng

Apache Spark / PySpark

Python

Scala

Hadoop / HDFS / Hive

ETL / ELT pipelines

SQL

Airflow

Git / CI/CD / Docker

Snowflake / Teradata / SQL Server

Kafka / real-time streaming

Spark SQL / HiveQL / SQL

Tools

Azure Databricks

Azure Data Factory

AWS EMR

Apache Airflow

Job description

UARROW PTE. LTD. is seeking an experienced Data Engineer to design, build and optimize scalable data pipelines for large datasets. You will work across Spark/PySpark, Python/Scala, and cloud services to deliver reliable ingestion, transformation and warehousing capabilities.

You will deploy batch and real-time processing using Spark Structured Streaming and Kafka, collaborate with cross‑functional teams, and ensure data quality, security and cost efficiency in a fast-paced enterprise environment.