Data Engineer

UARROW PTE. LTD.; Singapore; On-site; SGD 110,000 - 190,000; Full time

Job summary

UARROW PTE. LTD. is seeking an experienced Data Engineer to design, build and optimize scalable data pipelines for large datasets. You will work across Spark/PySpark, Python/Scala, and cloud services to deliver reliable ingestion, transformation and warehousing capabilities.

You will deploy batch and real-time processing using Spark Structured Streaming and Kafka, collaborate with cross‑functional teams, and ensure data quality, security and cost efficiency in a fast-paced enterprise environment.

Qualifications

• 6+ years of experience in Data Engineering / Big Data Eng.
• Strong hands‑on with Apache Spark / PySpark.
• Strong programming in Python and/or Scala.
• Good experience with Hadoop, HDFS and Hive.
• Experience developing ETL/ELT and data ingestion pipelines.
• Strong SQL and data processing skills.
• Experience with Azure Databricks, Azure Data Factory, AWS EMR.
• Experience with Kafka / real‑time streaming is an advantage.
• Hands‑on experience with Airflow and data pipeline orchestration.
• Experience with Snowflake, Teradata, SQL Server or other enterprise databases.

Responsibilities

• Design, develop and maintain scalable data pipelines and ingestion frameworks for large-volume datasets.
• Develop data transformation and processing applications using Spark, PySpark, Scala and Python.
• Build and optimize pipelines using Azure Databricks, Azure Data Factory, AWS EMR and related clouds.
• Work with Hadoop, HDFS, Hive, Snowflake, Teradata and Data Lake environments.
• Develop batch and real-time data processing using Spark Structured Streaming and Kafka.
• Perform data extraction, transformation and loading across heterogeneous sources.
• Develop and optimize Spark SQL, HiveQL and SQL queries for performance and cost efficiency.
• Design data models, partitioning strategies and scalable storage architectures.
• Build and manage workflow orchestration using Apache Airflow.
• Implement CI/CD and automated testing using Jenkins, Docker, GitHub Actions and pytest.
• Troubleshoot data pipelines and production issues; deliver sustainable solutions.
• Collaborate with stakeholders to understand requirements and deliver data engineering solutions.
• Ensure data quality, reliability, security and operational stability across platforms.

Skills

Data Engineering / Big Data Eng

Apache Spark / PySpark

Python

Scala

Hadoop / HDFS / Hive

ETL / ELT pipelines

SQL

Airflow

Git / CI/CD / Docker

Snowflake / Teradata / SQL Server

Kafka / real-time streaming

Spark SQL / HiveQL / SQL

Tools

Azure Databricks

Azure Data Factory

AWS EMR

Apache Airflow

Job description

Key Responsibilities

• Design, develop and maintain scalable data pipelines and data ingestion frameworks for large-volume datasets.
• Develop data transformation and processing applications using Apache Spark, PySpark, Scala and Python.
• Build and optimize data pipelines using Azure Databricks, Azure Data Factory, AWS EMR and related cloud services.
• Work with Hadoop, HDFS, Hive, Snowflake, Teradata and Data Lake environments.
• Develop batch and real-time data processing solutions using Spark Structured Streaming and Kafka.
• Perform data extraction, transformation and loading across heterogeneous source and target systems.
• Develop and optimize Spark SQL, HiveQL and SQL queries for performance and cost efficiency.
• Design data models, partitioning strategies and scalable data storage architectures.
• Build and manage workflow orchestration using Apache Airflow.
• Implement CI/CD pipelines and automated testing using tools such as Jenkins, Docker, GitHub Actions and pytest.
• Troubleshoot data pipeline, performance and production issues and implement sustainable solutions.
• Collaborate with business stakeholders, architects and technology teams to understand requirements and deliver data engineering solutions.
• Ensure data quality, reliability, security and operational stability across enterprise data platforms.

Required Skills

• 6+ years of experience in Data Engineering / Big Data Engineering.
• Strong hands‑on experience with Apache Spark / PySpark.
• Strong programming skills in Python and/or Scala.
• Good experience with Hadoop, HDFS and Hive.
• Experience developing ETL/ELT and data ingestion pipelines.
• Strong SQL and data processing skills.
• Experience with Azure Databricks, Azure Data Factory, AWS EMR or equivalent cloud data platforms.
• Experience with Kafka / real‑time streaming is an advantage.
• Hands‑on experience with Airflow and data pipeline orchestration.
• Experience with Snowflake, Teradata, SQL Server or other enterprise databases.
• Good understanding of Data Lake, Delta Lake, Data Warehousing and Data Modelling.
• Experience with Git, CI/CD, Docker and automated testing.