Senior Data Infra Engineer, LLM Pre-Training

Anthropic Limited; San Francisco (CA); Hybrid; USD 500,000 - 850,000; Full time

Benefits offered by this job

Equity donation matching

Generous vacation and parental leave

Flexible working hours

Office space in SF

Job summary

Anthropic is seeking a Staff-level Engineer for the Pre-training team in San Francisco to design and build data processing pipelines for large language model training. You will work at the intersection of research and production, delivering scalable, fault-tolerant systems and novel data processing architectures.

Ideal candidates have 5+ years of software engineering experience, strong distributed systems skills, and hands-on Spark, Python, and Rust experience.

Qualifications

• 5+ years of software engineering experience outside internships.
• Strong skills building high-throughput distributed systems.
• Hands-on experience with distributed computing frameworks, especially Apache Spark.
• Excellent problem-solving skills and attention to detail.
• Strong communication and collaboration abilities.
• Advanced degree in Computer Science or related field.
• Experience with language model training infrastructure.
• Background in Data Infrastructure, MLOps, or ML infrastructure.

Responsibilities

• Design and implement data processing infrastructure for large language model training (highly performant, reproducible, traceable)
• Develop and maintain core processing primitives (e.g., tokenization, deduplication, chunking) with a focus on scalability
• Build robust systems for data quality assurance and validation at scale
• Collaborate with research teams to implement novel data processing architectures
• Build and operate end-to-end data pipelines that turn raw web-scale corpora into training-ready datasets

Skills

Distributed systems

Python

Rust

Communication

Education

Advanced degree in Computer Science

Tools

Apache Spark

Job description

Anthropic is seeking a Staff-level Engineer for the Pre-training team in San Francisco to design and build data processing pipelines for large language model training. You will work at the intersection of research and production, delivering scalable, fault-tolerant systems and novel data processing architectures.

Ideal candidates have 5+ years of software engineering experience, strong distributed systems skills, and hands-on Spark, Python, and Rust experience.