Data Infrastructure Engineer for Large-Scale AI Pipelines

Thinking Machines Lab Inc.; San Francisco (CA); On-site; USD 300,000 - 400,000; Full time

Benefits offered by this job

Health benefits

Dental & Vision benefits

Unlimited PTO

Paid parental leave

Relocation support

Job summary

Thinking Machines Lab Inc. in San Francisco is seeking an engineer to help build and scale the core data infrastructure for distributed training pipelines and multimodal data catalogs.

You will work with researchers to accelerate experiments, develop datasets, and improve reliability across LLM research platforms, using Spark, Kafka, Beam, Ray, and Delta Lake.

This role emphasizes end-to-end ownership, collaboration across teams, and building from the ground up in a fast-moving environment.

Qualifications

• Bachelor's degree or equivalent experience in CS or related field.
• Proficiency in at least one backend language (Python or Rust).
• Fluency with distributed compute frameworks such as Apache Spark or Ray.
• Deep familiarity with cloud infrastructure, data lake architectures, and batch/streaming pipelines.
• Comfort operating across the stack and owning projects end-to-end.

Responsibilities

• Design, build, and operate scalable, fault-tolerant infrastructure for LLM Research: distributed compute, data orchestration, and storage across modalities.
• Develop high-throughput systems for data ingestion, processing, and transformation—training data catalogs, deduplication, quality checks, and search.
• Build systems for traceability, reproducibility, and robust quality control at every stage of the data lifecycle.
• Implement and maintain monitoring and alerting to support platform reliability and performance.
• Collaborate with research teams to unlock new features, improve data quality, and accelerate training cycles.

Skills

Python

Rust

Apache Spark

Ray

Cloud infrastructure

Education

Bachelor's degree or equivalent experience

Tools

Kafka

dbt

Terraform

Airflow

Job description

Thinking Machines Lab Inc. in San Francisco is seeking an engineer to help build and scale the core data infrastructure for distributed training pipelines and multimodal data catalogs.

You will work with researchers to accelerate experiments, develop datasets, and improve reliability across LLM research platforms, using Spark, Kafka, Beam, Ray, and Delta Lake.

This role emphasizes end-to-end ownership, collaboration across teams, and building from the ground up in a fast-moving environment.