Senior Data Platform Engineer — On-Prem/Hybrid, Kafka & Ceph

PATHSOURCE CONSULTING PTE. LTD.; Singapore; Sur place; SGD 120 000 - 180 000; Plein temps

Résumé du poste

PATHSOURCE CONSULTING PTE. LTD. in Singapore seeks a hands‑on Senior Infrastructure Engineer to deploy and operate the ingestion and distributed storage layer of a large‑scale on‑premise data platform.

The role is a six‑month contract with potential extension or conversion based on performance and business needs. The candidate will design, deploy and run scalable ingestion services using Kafka and related frameworks, containerise components with Docker, and operate Ceph storage in on‑premise or

Qualifications

• Minimum six years of hands‑on experience in data‑platform infrastructure, distributed storage, data ingestion or a closely related role.
• Production experience with both Docker and Kubernetes is mandatory.
• Strong experience deploying and operating data platforms in on‑premise or hybrid environments.
• Hands‑on production experience with Apache Kafka and distributed ingestion architecture.
• Practical experience deploying or administering Ceph or comparable software‑defined distributed storage.
• Experience with Apache Iceberg and data catalogues such as Apache Polaris.
• Experience using dltHub or a comparable Python‑based data‑ingestion framework.
• Strong understanding of Linux, networking, storage, high availability, performance tuning, backup and recovery.
• Experience supporting high‑volume platforms at terabyte or petabyte scale.
• Ability to operate independently and communicate effectively with client and technical stakeholders.
• Willingness to accept a six‑to‑twelve‑month contract and attend client locations when required.

Responsabilités

• Design, deploy and operate scalable batch and streaming ingestion services in an on‑premise or hybrid environment.
• Build and support high‑throughput data ingestion using Apache Kafka and dltHub or comparable ingestion frameworks.
• Containerise platform components using Docker and deploy and operate production workloads on Kubernetes.
• Deploy and administer Ceph‑based distributed storage, covering capacity, performance, availability and recovery.
• Build and maintain lakehouse storage using Apache Iceberg tables and Apache Polaris catalogues.
• Integrate ingestion and storage services with downstream Spark, Trino, ClickHouse and transformation workloads.
• Monitor platform health, throughput, storage and logs using VictoriaMetrics and VictoriaLogs or comparable tools.
• Troubleshoot ingestion, storage, networking, Kubernetes and production‑performance issues.
• Produce technical documentation, operating procedures and recovery runbooks.
• Work directly with technical stakeholders and contribute as a senior individual contributor.

Connaissances

Data‑platform infrastructure

Distributed storage

Data ingestion

Docker

Kubernetes

Apache Kafka

Ceph

Apache Iceberg

Apache Polaris

dltHub

Linux

Networking

Storage

High availability

Performance tuning

Backup and recovery

Outils

Docker

Kubernetes

Apache Kafka

Ceph

Apache Iceberg

Apache Polaris

dltHub

Spark

Trino

ClickHouse

dbt

Description du poste

PATHSOURCE CONSULTING PTE. LTD. in Singapore seeks a hands‑on Senior Infrastructure Engineer to deploy and operate the ingestion and distributed storage layer of a large‑scale on‑premise data platform.

The role is a six‑month contract with potential extension or conversion based on performance and business needs. The candidate will design, deploy and run scalable ingestion services using Kafka and related frameworks, containerise components with Docker, and operate Ceph storage in on‑premise or