Staff Data Infrastructure Engineer - Petabyte-Scale Platform

Causal; San Francisco (CA); On-site; USD 150,000 - 190,000; Full time

Job summary

Causal is building a Large Physics foundation Model and seeks a data engineer to design the data platform that underpins this effort, enabling cheap ingestion, fast querying, and reliable training. You will design petabyte-scale storage, manage a shared compute and orchestration layer, and optimize end-to-end data loading from raw sources to tensor-ready formats.

Join a mission-driven team applying data engineering at scale to weather and physical systems, with opportunities to shape system

Qualifications

• Experience building large-scale data pipelines and distributed compute systems (Spark, Ray, Beam).
• Knowledge of data ingestion, storage, and loading — including file formats and storage systems (Parquet, Zarr, Delta Lake) and how they impact performance and scalability.
• Deep familiarity with cloud infrastructure, data lake architectures, and batch and streaming pipelines.
• Understanding of how data loading throughput affects large-scale training and how to optimize it.
• Owns deliverables end-to-end, from collecting and translating requirements to autonomously driving execution.

Responsibilities

• Design and operate petabyte-scale storage: lakehouse architecture, file formats, and data layout optimized for both batch and real-time queries.
• Own the shared compute and orchestration platform (e.g. Spark, Ray, workflow scheduling) that ingestion and research pipelines run on.
• Optimize data strategy end to end from storage to loading, owning high-throughput data loading into training up to the tensor boundary.
• Build systems for cataloging, deduplication, lineage, search, and reproducibility at every stage of the data lifecycle.
• Implement the platform-level quality and monitoring tooling that data and research teams build their checks on.
• Scale infrastructure to improve engineering velocity and ensure reliability, with monitoring and alerting to match.
• Work across the full data lifecycle when the mission needs it — including building and operating ingestion pipelines for critical data sources directly.

Skills

Spark

Ray

Beam

Data pipelines

Cloud infra

Parquet

Zarr

Delta Lake

Tools

Parquet

Zarr

Delta Lake

Job description

Causal is building a Large Physics foundation Model and seeks a data engineer to design the data platform that underpins this effort, enabling cheap ingestion, fast querying, and reliable training. You will design petabyte-scale storage, manage a shared compute and orchestration layer, and optimize end-to-end data loading from raw sources to tensor-ready formats.

Join a mission-driven team applying data engineering at scale to weather and physical systems, with opportunities to shape system