Staff Data Infrastructure Engineer (Petabyte-scale)

Causal Labs; San Francisco (CA); On-site; USD 150,000 - 210,000; Full time

Job summary

Causal Labs is building a Large Physics foundation Model and seeks a data platform engineer to design and operate its petabyte-scale storage and processing systems. You will own the shared compute platform and end-to-end data loading for training, enabling high-throughput pipelines and reliable data workflows.

You will work with state-of-the-art data lake tech and cloud infrastructure, shaping data ingestion strategies and ensuring reproducibility across experiments in a fast-paced research

Qualifications

• Experience building large-scale data pipelines and distributed compute systems (e.g. Spark, Ray, Beam).
• Knowledge of data ingestion, storage, and loading—file formats and storage systems like Parquet, Zarr, Delta Lake.
• Deep familiarity with cloud infrastructure and data lake architectures.

Responsibilities

• Design and operate petabyte-scale storage: lakehouse architecture, file formats, and data layout for batch and real-time queries.
• Own the shared compute and orchestration platform used by ingestion and research pipelines.
• Optimize data strategy end-to-end from storage to loading, enabling high-throughput training data flows.
• Build cataloging, deduplication, lineage, search, and reproducibility tooling across the data lifecycle.
• Implement platform-level quality and monitoring tooling for data and research teams.
• Scale infrastructure to improve engineering velocity with reliable monitoring and alerting.
• Collaborate across the full data lifecycle to ingest critical data sources as needed.

Skills

Large-scale data pipelines

Distributed compute systems

Cloud infrastructure

Data ingestion optimization

Data lake architectures

Tools

Spark

Ray

Beam

Parquet

Zarr

Delta Lake

Job description

Causal Labs is building a Large Physics foundation Model and seeks a data platform engineer to design and operate its petabyte-scale storage and processing systems. You will own the shared compute platform and end-to-end data loading for training, enabling high-throughput pipelines and reliable data workflows.

You will work with state-of-the-art data lake tech and cloud infrastructure, shaping data ingestion strategies and ensuring reproducibility across experiments in a fast-paced research