Senior Data Engineer - Batch Ingestion and Data Reliability (Contract) - URGENT HIRING

PATHSOURCE CONSULTING PTE. LTD.; Singapore; On-site; SGD 112,000 - 167,000; Full time

Job summary

PATHSOURCE CONSULTING PTE. LTD. in Singapore seeks a Senior Data Engineer to own batch ingestion, backfills and CDC for an on-premises data platform built for a regional operator. You will ensure complete, accurate telemetry history and support production readiness for APAC sites.

The role is a six-month contract with potential extension depending on project needs; office-based in Singapore with travel to client sites, and immediate availability preferred for quick start.

Qualifications

• 4 to 7 or more years of experience building and operating production data pipelines.
• Strong hands-on Python and SQL skills.
• Production experience ingesting from APIs and relational databases, including incremental loading and change data capture.
• Demonstrated delivery of large backfills or data migrations with source-to-target reconciliation.
• Experience building idempotent pipelines that handle deduplication, partition overwrites and late-arriving data.
• Production Spark experience for large batch workloads.
• Experience designing tables in Apache Iceberg, Delta Lake or Apache Hudi, including partitioning and schema evolution.
• Production on-call and incident-management experience, including incident documentation.
• Working knowledge of an orchestrator such as Dagster, Airflow or Prefect, together with Linux, Docker and Git.

Responsibilities

• Backfill 12 months of telemetry history from the existing monitoring platform and reconcile the results to confirm that no records were lost.
• Build a replay process that restores data following an outage, supported by a three-hour streaming buffer.
• Develop batch and change data capture pipelines from business systems covering contracts, costs, labour and related operational data, using dlt or comparable tools.
• Work with source-system teams to define data contracts covering schemas, delivery frequency and failure conditions.
• Create pipeline-health dashboards and automated responses for freshness, volume, quality and failure issues.
• Load historical data as new sites come online and support monthly production releases.
• Participate in the production on-call rotation, investigate incidents and document root causes and recovery steps.
• Pair with the client's data engineers and prepare clear runbooks for operational handover.

Skills

Python

SQL

API ingestion

CDC

Backfills

Data migrations

Idempotent pipelines

Spark

Iceberg/Delta/Hudi

Orchestrators

Linux

Docker

Git

Tools

Dagster

Airflow

Prefect

Kubernetes

Trino

MinIO

Ceph

Prometheus

VictoriaMetrics

Airbyte

Debezium

Kafka Connect

Job description

The client is a Singapore-based data consultancy building anon-premises data platform for a major data-centre operator across APAC. The platform will retain complete telemetry history for each site and combine itwith business data such as contracts, costs and labour.

This Senior Data Engineer will own batch ingestion, historicalbackfills, change data capture and recovery processes, ensuring the platform'shistory remains complete, accurate and supportable in production.

This is an initial six-month contract. Any extension will depend onproject requirements and is not guaranteed.

Immediate Availability

Immediate hiring priority: Priority will be given to Singapore Citizens and Permanent Residents who can start immediately or within two weeks. The role is best suited to candidates between roles, nearing the end of a contract or project, or otherwise available at short notice. Extended notice periods are unlikely to meet the project timeline.

Key Responsibilities

• Backfill 12 months of telemetry history from the existing monitoring platform and reconcile the results to confirm that no records were lost.
• Build a replay process that restores data following an outage, supported by a three-hour streaming buffer.
• Develop batch and change data capture pipelines from business systems covering contracts, costs, labour and related operational data, using dlt or comparable tools.
• Work with source-system teams to define data contracts covering schemas, delivery frequency and failure conditions.
• Create pipeline-health dashboards and automated responses for freshness, volume, quality and failure issues.
• Load historical data as new sites come online and support monthly production releases.
• Participate in the production on-call rotation, investigate incidents and document root causes and recovery steps.
• Pair with the client's data engineers and prepare clear runbooks for operational handover.

Requirements

• 4 to 7 or more years of experience building and operating production data pipelines.
• Strong hands-on Python and SQL skills.
• Production experience ingesting from APIs and relational databases, including incremental loading and change data capture.
• Demonstrated delivery of large backfills or data migrations with source-to-target reconciliation.
• Experience building idempotent pipelines that handle deduplication, partition overwrites and late-arriving data.
• Production Spark experience for large batch workloads.
• Experience designing tables in Apache Iceberg, Delta Lake or Apache Hudi, including partitioning and schema evolution.
• Production on-call and incident-management experience, including incident documentation.
• Working knowledge of an orchestrator such as Dagster, Airflow or Prefect, together with Linux, Docker and Git.

Preferred Experience

• Experience running data systems on-premises or on self-managed infrastructure, including Kubernetes, self-hosted S3-compatible storage such as MinIO or Ceph, and distributed query engines such as Trino.
• Experience handling sensitive personnel or commercial data and enforcing data contracts or upstream schemas.
• Experience exporting or restoring high-volume time-series data from systems such as Prometheus or VictoriaMetrics.
• Exposure to dlt, Airbyte, Debezium or Kafka Connect.

Contract and Work Arrangement

• Initial six-month contract with the Singapore-based data consultancy.
• Open to Singapore Citizens and Permanent Residents only.
• Based in Singapore, with office attendance expected three to five days per week.
• Travel to client sites in Australia or elsewhere in the region may be required for site go-lives.
• Participation in the production on-call rotation is required and may involve work outside normal business hours.

Hiring Process

Please note: The hands-on technical assessment is a mandatory selection stage.

1. Technical assessment and interview based on a practical problem drawn from the role.
2. Team interview covering engineering approach, code review, delivery and incident handling.
3. Client interview with the client's engineering team.

For Immediate Consideration

Please indicate the following information in your resume:

• Current and expected salary
• Reason or reasons for leaving
• Earliest available commencement date
• Confirmation that you are prepared to complete the mandatory technical assessment

By submitting an application or resume to us, you consent to PathSource Consulting collecting, using, retaining and disclosing your personal information to prospective employers for recruitment purposes.