Data Infrastructure Engineer - Large-Scale ML Pipelines

Visa Hunt; San Francisco (CA); On-site; USD 350,000 - 475,000; Full time

Benefits offered by this job

Health, dental, and vision benefits

Unlimited PTO

Parental leave

Relocation support

Job summary

Thinking Machines in San Francisco is seeking an engineer to join our data infrastructure team. You will help architect and scale core infrastructure behind distributed training pipelines, multimodal data catalogs, and processing systems handling petabytes of data.

You'll collaborate with researchers to accelerate experiments, build high-throughput data ingestion and quality checks, and ensure traceability and reliability across the data lifecycle.

Qualifications

• Bachelor’s degree or equivalent in computer science, engineering, or similar.
• Proficiency in at least one backend language (Python or Rust).
• Experience with distributed compute frameworks (Spark or Ray).
• Familiarity with cloud infrastructure, data lake architectures, and batch and streaming pipelines.
• Ability to own projects end-to-end.
• Strong collaboration across cross-functional teams.
• Bias for action and initiative to work across stacks to ship features.

Responsibilities

• Design, build, and operate scalable, fault-tolerant infrastructure for LLM Research: distributed compute, data orchestration, and storage across modalities.
• Develop high-throughput systems for data ingestion, processing, and transformation — including training data catalogs, deduplication, quality checks, and search.
• Build systems for traceability, reproducibility, and robust quality control at every stage of the data lifecycle.
• Implement and maintain monitoring and alerting to support platform reliability and performance.
• Collaborate with research teams to unlock new features, improve data quality, and accelerate training cycles.

Skills

Python

Rust

Distributed compute

Data pipelines

Cloud infrastructure

End-to-end ownership

Collaborative teamwork

Education

Bachelor’s degree in CS or engineering

Tools

Spark

Ray

Kafka

dbt

Terraform

Airflow

Job description

Thinking Machines in San Francisco is seeking an engineer to join our data infrastructure team. You will help architect and scale core infrastructure behind distributed training pipelines, multimodal data catalogs, and processing systems handling petabytes of data.

You'll collaborate with researchers to accelerate experiments, build high-throughput data ingestion and quality checks, and ensure traceability and reliability across the data lifecycle.