Member of Technical Staff — Data Infrastructure

Kindredventures; San Francisco (CA); On-site; USD 130,000 - 170,000; Full time

Job summary

Kindredventures is building a Large Physics foundation Model and seeks a data engineer to design and operate the data platform for petabyte-scale storage, real-time and batch queries, and scalable ingestion.

You will own the compute and orchestration platforms (Spark, Ray, Beam), optimize data loading to training, and implement cataloging, lineage, and monitoring across the data lifecycle. Collaboration across research and engineering is essential.

Qualifications

• Demonstrated experience building large-scale data pipelines and distributed compute systems (e.g. Spark, Ray, Beam)
• Knowledge of state-of-the-art data ingestion, storage, and loading — including file formats and storage systems (e.g. Parquet, Zarr, Delta Lake) and how they impact performance and scalability
• Deep familiarity with cloud infrastructure, data lake architectures, and batch and streaming pipelines
• Understanding of how data loading throughput affects large-scale training, and experience optimizing it
• Owns deliverables end-to-end, from collecting and translating requirements to autonomously driving execution

Responsibilities

• Design and operate petabyte-scale storage: lakehouse architecture, file formats, and data layout optimized for both batch and real-time queries
• Own the shared compute and orchestration platform (e.g. Spark, Ray, workflow scheduling) that ingestion and research pipelines run on
• Optimize data strategy end to end from storage to loading, owning high-throughput data loading into training up to the tensor boundary
• Build systems for cataloging, deduplication, lineage, search, and reproducibility at every stage of the data lifecycle
• Implement the platform-level quality and monitoring tooling that data and research teams build their checks on
• Scale infrastructure to improve engineering velocity and ensure reliability, with monitoring and alerting to match
• Work across the full data lifecycle when the mission needs it — including building and operating ingestion pipelines for critical data sources directly

Skills

Large-scale data pipelines

Distributed compute systems

Cloud infrastructure

Data ingestion best practices

Autonomous delivery ownership

Tools

Spark

Ray

Beam

Parquet

Zarr

Delta Lake

Job description

Our mission is general causal intelligence; AI that is capable of (1) predicting the future and (2) identifying the actions to alter it.

To achieve this breakthrough, we are building a Large Physics foundation Model (LPM) because physical systems, unlike text or images, are governed by verifiable cause and effect. We believe that scaling on physics will enable an understanding of causality required to predict and control physical systems, starting with weather.

Our founding team has built and deployed AI against the physical world in robotics, drug discovery, and particle physics at institutions like DeepMind, Waymo, Cruise, Insitro, Nabla Bio, and CERN.

We look for data engineers who are excited to tackle unsolved problems. Physical observations arrive continuously, in many formats, at a scale that dwarfs what is used to train today's LLMs. Your mission is to build the data platform underneath it all — the storage, compute, and loading systems that make every dataset cheap to ingest, fast to query, and immediately available to training.

Responsibilities

• Design and operate petabyte-scale storage: lakehouse architecture, file formats, and data layout optimized for both batch and real-time queries

• Own the shared compute and orchestration platform (e.g. Spark, Ray, workflow scheduling) that ingestion and research pipelines run on

• Optimize data strategy end to end from storage to loading, owning high-throughput data loading into training up to the tensor boundary

• Build systems for cataloging, deduplication, lineage, search, and reproducibility at every stage of the data lifecycle

• Implement the platform-level quality and monitoring tooling that data and research teams build their checks on

• Scale infrastructure to improve engineering velocity and ensure reliability, with monitoring and alerting to match

• Work across the full data lifecycle when the mission needs it — including building and operating ingestion pipelines for critical data sources directly

What we're looking for

We value a relentless approach to problem-solving, rapid execution, and the ability to quickly learn in unfamiliar domains.

• Demonstrated experience building large-scale data pipelines and distributed compute systems (e.g. Spark, Ray, Beam)

• Knowledge of state-of-the-art methods and tools for data ingestion, storage, and loading — including file formats and storage systems (e.g. Parquet, Zarr, Delta Lake) and how they impact performance and scalability

• Deep familiarity with cloud infrastructure, data lake architectures, and batch and streaming pipelines

• Understanding of how data loading throughput affects large-scale training, and experience optimizing it

• Owns deliverables end-to-end, from collecting and translating requirements to autonomously driving execution