Senior Data Engineer - Streaming Pipelines, Kafka & Iceberg

PATHSOURCE CONSULTING PTE. LTD.; Singapore; Sur place; SGD 90 000 - 130 000; Plein temps

Résumé du poste

PATHSOURCE CONSULTING PTE. LTD. is seeking a Senior Data Engineer in Singapore to build streaming pipelines and delivery tooling for a major APAC data-center operator.

The role focuses on Kafka-to-Iceberg pipelines, offset management, and scaling telemetry ingestion across teams. You will run Spark workloads in production for streaming and batch workloads, design Iceberg/Delta/Hudi tables, and implement dbt and orchestration tools.

Qualifications

• Four to seven or more years of experience building and operating production data pipelines.
• Strong hands-on Python and SQL skills.
• Production experience with Kafka consumers or Kafka Connect, including offset management and failure recovery.
• Experience running and tuning Spark jobs in production for streaming or batch workloads.
• Experience designing tables in Apache Iceberg, Delta Lake or Apache Hudi, including partitioning, file sizing, compaction and schema evolution.

Responsabilités

• Build and operate Kafka-to-Apache Iceberg streaming pipelines, beginning with alert history and documents and scaling across telemetry topics at three times current volume.
• Design reliable Kafka consumers or Kafka Connect integrations with correct offset management, replay and failure recovery.
• Run and tune Spark workloads in production across streaming and batch use cases.
• Design and optimise Iceberg tables covering partitioning, file sizing, compaction and schema evolution.
• Set up and maintain the team's data tooling, including dbt, Dagster or a comparable orchestrator, CI/CD, data-quality checks, lineage and documentation.
• Build feature tables and training datasets for the client's machine-learning team.
• Monitor data freshness, volume, performance and failures, and participate in the production on-call rotation.
• Pair with the client's data engineers and prepare clear runbooks for operational handover.

Connaissances

Python

SQL

Incident management

Team collaboration

Outils

Kafka

Kafka Connect

Spark

Apache Iceberg

Delta Lake

Apache Hudi

dbt

Dagster

Airflow

Prefect

CI/CD

Docker

Git

Linux

Description du poste

PATHSOURCE CONSULTING PTE. LTD. is seeking a Senior Data Engineer in Singapore to build streaming pipelines and delivery tooling for a major APAC data-center operator.

The role focuses on Kafka-to-Iceberg pipelines, offset management, and scaling telemetry ingestion across teams. You will run Spark workloads in production for streaming and batch workloads, design Iceberg/Delta/Hudi tables, and implement dbt and orchestration tools.