Staff Data Infrastructure Engineer, ML Training

Anthropic Limited; San Francisco, Northern (CA, KY); Hybrid; USD 510,000 - 850,000; Full time

Benefits offered by this job

Competitive compensation

Equity program (optional)

Generous vacation & parental leave

Flexible working hours

San Francisco office

Job summary

Anthropic is seeking a Staff Engineer to join the Pre-training team in San Francisco. You will design and optimize data processing infrastructure for large language model training, build scalable primitives, and ensure data quality at web scale.

You will collaborate with researchers to deploy new architectures, own end-to-end data pipelines, and help advance safe, steerable AI. A strong background in distributed systems and experience with Spark, Python, and ML infrastructure are essential.

Qualifications

• 5+ years of professional experience outside internships.
• Strong software engineering skills with experience building high-throughput fault-tolerant distributed systems.
• Hands-on experience with distributed computing frameworks, particularly Spark.
• Excellent problem-solving skills and attention to detail.
• Strong communication skills and ability to work in a collaborative environment.
• Advanced degree in Computer Science or related field.
• Experience with language model training infrastructure.
• Background in Data Infrastructure, MLOps, or ML infrastructure.

Responsibilities

• Design and implement data processing infrastructure for large language model training (highly performant, reproducible, traceable).
• Develop and maintain core processing primitives (tokenization, deduplication, chunking) with a focus on scalability.
• Build robust systems for data quality assurance and validation at scale.
• Collaborate with research teams to implement novel data processing architectures.
• Build and operate end-to-end data pipelines that turn raw web-scale corpora into training-ready datasets.

Skills

Distributed systems

Python

Rust

Spark

Problem solving

Communication

ML infrastructure

Leadership

Education

Bachelor's degree in Computer Science or related field

Tools

Apache Spark

Python

Rust

Job description

Anthropic is seeking a Staff Engineer to join the Pre-training team in San Francisco. You will design and optimize data processing infrastructure for large language model training, build scalable primitives, and ensure data quality at web scale.

You will collaborate with researchers to deploy new architectures, own end-to-end data pipelines, and help advance safe, steerable AI. A strong background in distributed systems and experience with Spark, Python, and ML infrastructure are essential.