Senior Data Infrastructure Engineer (Spatial Data Platform)

Dada Consultants Pte Ltd; Singapore; On-site; SGD 180,000 - 260,000; Full time

Job summary

Dada Consultants Pte Ltd is hiring a hands-on Senior Data Infrastructure Engineer to build and scale a petabyte-scale data lakehouse powering our spatial data products. You will design storage layouts, implement data versioning, and run large-scale processing pipelines in production.

In this role you will integrate ML training data pipelines, optimize cross-region transfers, and maintain IaC on AWS. You will work with Spark, Ray, Flink, Iceberg/Delta Lake, and LakeFS to deliver robust data

Qualifications

• 6+ years in data infrastructure or distributed systems engineering.
• Proven live petabyte-scale data platform experience with concrete throughput and failure modes.
• Python plus Go/Rust/C++/Java/Scala proficiency.
• Iceberg/Delta Lake and LakeFS experience.
• Spark, Ray or Flink processing; Airflow/Argo/Prefect orchestration.
• AWS (S3, EKS/ECS) and Terraform experience.

Responsibilities

• PB-Scale Lakehouse: design object storage layouts, partitions, catalogs, and open formats (Iceberg/Delta).
• Data Versioning & Ingest: build dataset lineage (LakeFS) and high-throughput ingest pipelines.
• Distributed CPU/GPU Compute: operate Spark, Ray, Flink for batch jobs and SLAM/3D processing.
• ML Training Integration: manage GPU clusters with WebDataset, Mosaic Streaming, and cross-region transfers.
• Curation & Querying: build indexing and fast SQL layers (Trino, DuckDB, Spark SQL) for researchers.
• Reliability & Infrastructure: maintain Terraform IaC on AWS and handle backfills/recovery.

Skills

Python

Go

Rust

C++

Java

Scala

Tools

Iceberg/Delta Lake

LakeFS

Glue/Nessie

Spark

Flink

Airflow

Argo

Prefect

Terraform

AWS S3

EKS/ECS

Kubernetes

Job description

Our client is looking for a hands-on Senior Data Infrastructure Engineer to build, scale, and operate the petabyte-scale data lakehouse behind our spatial data products.

Every field and lab capture—robotics, egocentric headsets, multi-camera video, depth, IMU—lands in this platform, gets versioned, and powers downstream 3D reconstructions (Gaussian Splats, SLAM) and ML training clusters.

This is a true engineering role. We need someone who has personally built and run a PB-scale platform in production, not just used one.

Key Responsibilities

• PB-Scale Lakehouse: Design object storage layouts (S3), partitioning strategies, catalog architectures, and open table formats (Iceberg/Delta).
• Data Versioning & Ingest: Build reproducible dataset lineage (LakeFS) and high-throughput ingest pipelines (validation, checksumming, deduplication, bronze $→$ gold promotion).
• Distributed CPU/GPU Compute: Operate large-scale processing (Spark, Ray, Flink) for frame extraction, sync/calibration, SLAM/3DGS batch jobs, and auto-labeling.
• ML Training Integration: Keep GPU clusters fed using sharded training formats (WebDataset, Mosaic Streaming), caching, and optimized cross-region transfers.
• Curation & Querying: Build indexing, search, and high-performance SQL query layers (Trino, DuckDB, Spark SQL) so researchers can slice data without copying it.
• Reliability & Infrastructure: Maintain IaC (Terraform) on AWS, handle pipeline backfills/recovery, optimize storage/egress costs, and participate in on-call rotation.

Qualifications

• Experience: 6+ years in data infrastructure or distributed systems engineering.
• Scale: Proven track record setting up and operating a live petabyte-scale data platform (able to speak concretely to throughput, object limits, and failure modes).
• Languages: Python + at least one of Go, Rust, C++, Java, or Scala.
• Table Formats & Versioning: Iceberg/Delta Lake, Catalogs (Glue/Nessie), LakeFS or content-addressed storage.
• Processing & Orchestration: Spark, Ray, or Flink; Airflow, Argo, or Prefect.
• Cloud/IaC: Strong AWS (S3, EKS/ECS, IAM) and Terraform experience.

Nice to Have

• Experience with unstructured/multimodal spatial data (video, LiDAR, point clouds, ROS bags/MCAP).
• GPU-accelerated pipelines (NVIDIA DALI, NVDEC, RAPIDS) or ML data loaders (WebDataset, Petastorm).
• Query engines (Trino, DuckDB, ClickHouse) and embedding/vector search.