Data Infra Engineer for Scalable AI Pipelines Unlimited PTO

AI Chopping Block, Inc.; San Francisco (CA); On-site; USD 300,000 - 400,000; Full time

Job summary

Thinking Machines in San Francisco, CA, is hiring an engineer to architect and scale core data infrastructure for distributed training and multimodal data catalogs. You’ll work with researchers to accelerate experiments and build from the ground up using Spark, Kafka, Beam, Ray, and Delta Lake.

Responsibilities include data ingestion, quality checks, deduplication, and search, plus end-to-end lifecycle traceability and robust monitoring to improve platform reliability.

Qualifications

• Bachelor's degree or equivalent in CS/engineering or related field.
• Proficiency in Python or Rust and at least one backend framework.
• Experience with distributed compute frameworks such as Spark or Ray.
• Strong understanding of cloud data lake architectures and pipelines.
• Ability to own projects end-to-end with cross-functional partners.

Responsibilities

• Design, build, and operate scalable infrastructure for LLM research.
• Develop high-throughput data ingestion, processing, and transformation pipelines.
• Create traceability and quality controls across the data lifecycle.
• Implement monitoring and alerting to ensure reliability and performance.
• Collaborate with researchers to speed up experiments and feature development.

Skills

Python

Rust

Apache Spark

Ray

Cloud infrastructure

Data lake architectures

Batch and streaming pipelines

End-to-end ownership

Cross-functional collaboration

initiative to ship

Education

Bachelor's degree in Computer Science or related field

Tools

Kafka

dbt

Terraform

Airflow

Job description

Thinking Machines in San Francisco, CA, is hiring an engineer to architect and scale core data infrastructure for distributed training and multimodal data catalogs. You’ll work with researchers to accelerate experiments and build from the ground up using Spark, Kafka, Beam, Ray, and Delta Lake.

Responsibilities include data ingestion, quality checks, deduplication, and search, plus end-to-end lifecycle traceability and robust monitoring to improve platform reliability.