Senior Data Engineer - 6-Month Contract, On-Prem Platform

PATHSOURCE CONSULTING PTE. LTD.; Singapore; On-site; SGD 112,000 - 167,000; Full time

Job summary

PATHSOURCE CONSULTING PTE. LTD. in Singapore seeks a Senior Data Engineer to own batch ingestion, backfills and CDC for an on-premises data platform built for a regional operator. You will ensure complete, accurate telemetry history and support production readiness for APAC sites.

The role is a six-month contract with potential extension depending on project needs; office-based in Singapore with travel to client sites, and immediate availability preferred for quick start.

Qualifications

• 4 to 7 or more years of experience building and operating production data pipelines.
• Strong hands-on Python and SQL skills.
• Production experience ingesting from APIs and relational databases, including incremental loading and change data capture.
• Demonstrated delivery of large backfills or data migrations with source-to-target reconciliation.
• Experience building idempotent pipelines that handle deduplication, partition overwrites and late-arriving data.
• Production Spark experience for large batch workloads.
• Experience designing tables in Apache Iceberg, Delta Lake or Apache Hudi, including partitioning and schema evolution.
• Production on-call and incident-management experience, including incident documentation.
• Working knowledge of an orchestrator such as Dagster, Airflow or Prefect, together with Linux, Docker and Git.

Responsibilities

• Backfill 12 months of telemetry history from the existing monitoring platform and reconcile the results to confirm that no records were lost.
• Build a replay process that restores data following an outage, supported by a three-hour streaming buffer.
• Develop batch and change data capture pipelines from business systems covering contracts, costs, labour and related operational data, using dlt or comparable tools.
• Work with source-system teams to define data contracts covering schemas, delivery frequency and failure conditions.
• Create pipeline-health dashboards and automated responses for freshness, volume, quality and failure issues.
• Load historical data as new sites come online and support monthly production releases.
• Participate in the production on-call rotation, investigate incidents and document root causes and recovery steps.
• Pair with the client's data engineers and prepare clear runbooks for operational handover.

Skills

Python

SQL

API ingestion

CDC

Backfills

Data migrations

Idempotent pipelines

Spark

Iceberg/Delta/Hudi

Orchestrators

Linux

Docker

Git

Tools

Dagster

Airflow

Prefect

Kubernetes

Trino

MinIO

Ceph

Prometheus

VictoriaMetrics

Airbyte

Debezium

Kafka Connect

Job description

PATHSOURCE CONSULTING PTE. LTD. in Singapore seeks a Senior Data Engineer to own batch ingestion, backfills and CDC for an on-premises data platform built for a regional operator. You will ensure complete, accurate telemetry history and support production readiness for APAC sites.

The role is a six-month contract with potential extension depending on project needs; office-based in Singapore with travel to client sites, and immediate availability preferred for quick start.