Senior Big Data Engineer (Compute & Transformation) – Contract

PATHSOURCE CONSULTING PTE. LTD.; Singapore; On-site; SGD 120,000 - 180,000; Full time

Job summary

The client PATHSOURCE CONSULTING PTE. LTD. in Singapore seeks a hands-on Senior Infrastructure Engineer to deploy and operate the compute and transformation layer of a large-scale data platform on-premise or hybrid.

This initial contract role may extend or convert based on business needs and performance. You will design, deploy, and operate distributed compute and transformation services, containerise with Docker, and run workloads on Kubernetes, Spark, Trino and ClickHouse.

Qualifications

• Minimum six years of hands-on experience in data-platform, big-data infrastructure, distributed systems or a closely related role.

Responsibilities

• Design, deploy and operate distributed compute, query and transformation services in an on-premise or hybrid environment.
• Containerise platform components using Docker and deploy and operate production workloads on Kubernetes.
• Build and support large-scale processing and query services using Spark, Trino and ClickHouse.
• Develop and maintain governed transformation workflows using dbt.
• Integrate compute services with Apache Iceberg tables and Apache Polaris catalogues.
• Tune workload performance, resource allocation, concurrency and reliability for terabyte- or petabyte-scale data.
• Implement monitoring and logging using VictoriaMetrics and VictoriaLogs or comparable observability tools.
• Troubleshoot distributed-system, workload, storage-integration and production-performance issues.
• Produce technical documentation, operating procedures and deployment runbooks.
• Work directly with technical stakeholders and contribute as a senior individual contributor.

Skills

Data-platform infra

Docker

Kubernetes

Spark

Trino

ClickHouse

dbt

Linux

Networking

High availability

On-premise

Tools

Docker

Kubernetes

Spark

Trino

ClickHouse

dbt

Apache Iceberg

Apache Polaris

VictoriaMetrics

VictoriaLogs

Ceph

Job description

The client is a Singapore-based data consultancy seeking a hands-on Senior Infrastructure Engineer to deploy and operate the compute and transformation layer of a large-scale on-premise data platform.

This is an initial contract position, with the potential for extension or conversion to a permanent position based on business requirements and individual performance.

Key Responsibilities

• Design, deploy and operate distributed compute, query and transformation services in an on-premise or hybrid environment.
• Containerise platform components using Docker and deploy and operate production workloads on Kubernetes.
• Build and support large-scale processing and query services using Spark, Trino and ClickHouse.
• Develop and maintain governed transformation workflows using dbt.
• Integrate compute services with Apache Iceberg tables and Apache Polaris catalogues.
• Tune workload performance, resource allocation, concurrency and reliability for terabyte- or petabyte-scale data.
• Implement monitoring and logging using VictoriaMetrics and VictoriaLogs or comparable observability tools.
• Troubleshoot distributed-system, workload, storage-integration and production-performance issues.
• Produce technical documentation, operating procedures and deployment runbooks.
• Work directly with technical stakeholders and contribute as a senior individual contributor.

Requirements

• Minimum six years of hands-on experience in data-platform, big-data infrastructure, distributed systems or a closely related role.
• Production experience with both Docker and Kubernetes is mandatory.
• Strong experience deploying and operating distributed data workloads in on-premise or hybrid environments.
• Hands-on experience with distributed compute and query technologies such as Spark, Trino and ClickHouse.
• Practical experience with data transformation and lakehouse technologies including dbt, Apache Iceberg and Apache Polaris.
• Experience monitoring platform workloads through VictoriaMetrics, VictoriaLogs or comparable technologies.
• Strong understanding of Linux, networking, resource management, performance tuning, high availability and recovery.
• Experience supporting high-volume platforms at terabyte or petabyte scale.
• Ability to operate independently and communicate effectively with client and technical stakeholders.
• Willingness to accept a six-to-twelve-month contract and attend client locations when required.

Preferred Experience

• Data-platform delivery within regulated sectors such as banking, fintech, healthcare or data centres.
• Experience with Kubernetes Operators, Helm, CI/CD and infrastructure automation.
• Experience integrating compute engines with distributed storage such as Ceph.

This position focuses on the compute and transformation infrastructure of a data platform. It is not primarily a business-intelligence, dashboarding, visualisation or data-analyst role.