Eval Infrastructure Tech Lead — Scalable AI Evaluation

United States Digital Space LLC; San Francisco (CA); Hybrid; USD 500,000 - 850,000; Full time

Job summary

United States Digital Space LLC is seeking an experienced tech lead to own and scale the Evals Infrastructure that measures model safety and capabilities. You will oversee distributed systems, optimize throughput, and drive reproducible benchmarks that inform launch decisions.

You will work at the intersection of research, inference, and infrastructure, mentoring engineers and collaborating with researchers to translate evaluation needs into scalable, trustworthy systems.

Qualifications

• Experience leading large-scale distributed systems projects.
• Strong coding skills in Python and Rust.
• Ability to design measurable, reproducible evals and dashboards.
• Interest in safe, ethical AI and potential impact on society.

Responsibilities

• Lead the team building the distributed systems that schedule, orchestrate, and execute evals for frontier model training.
• Own eval throughput and cost: compute allocation across suites, queueing against constrained accelerator pools, caching and reuse of eval work.
• Build and scale the harnesses researchers use to define, run, and iterate on evals.
• Make eval results trustworthy — determinism, reproducibility, and honest uncertainty quantification on reported metrics.
• Ensure eval signal reaches the dashboards and reviews where launch decisions actually get made.
• Contribute directly as an engineer while managing and growing the team, prioritizing its work, and coaching your reports.

Skills

Python

Rust

Distributed systems

Measurement quality

Research collaboration

Safety focus

Education

Bachelor's degree

Job description

United States Digital Space LLC is seeking an experienced tech lead to own and scale the Evals Infrastructure that measures model safety and capabilities. You will oversee distributed systems, optimize throughput, and drive reproducible benchmarks that inform launch decisions.

You will work at the intersection of research, inference, and infrastructure, mentoring engineers and collaborating with researchers to translate evaluation needs into scalable, trustworthy systems.