Staff AI Evaluation & Benchmark Engineer

Magic; San Francisco (CA); On-site; USD 200,000 - 550,000; Full time

Benefits offered by this job

Equity

401(k) plan

Health, dental, and vision insurance

Unlimited paid time off

Visa sponsorship

Relocation support

Job summary

Magic in San Francisco is seeking a Member of Technical Staff for the Evals team to design and validate new evaluations and benchmarks across model training and deployment. You will investigate model behavior, measure data quality, and improve evaluation correctness and reproducibility.

Join a team focused on rigorous measurement, reproducibility, and reliability, building tools that make results accessible. Visa sponsorship and relocation support to SF are offered.

Qualifications

• Experience developing model evaluations, benchmarks, or ML experiments.
• Track record of owning technical projects end-to-end.
• Skepticism toward results that cannot be reproduced, validated, or explained.
• Experience analyzing model performance and interpreting experimental results.
• Comfortable navigating ambiguity and determining whether a measurement actually captures the behavior it claims to measure.
• Commitment to raising the standard of rigor in AI evaluations.

Responsibilities

• Design and validate new evaluations and benchmarks across model training and deployment.
• Investigate model behavior and identify gaps in existing evaluations.
• Measure dataset quality and identify opportunities to improve training data.
• Improve evaluation correctness, reproducibility, and reliability.
• Audit and improve upon public benchmarks, research papers, and open-source implementations.
• Define metrics that accurately reflect model quality.
• Build tools that make evaluation results accessible and useful across Magic.

Skills

Model evaluations

ML experiments

Programming skills

End-to-end ownership

Reproducibility focus

Data analysis

Ambiguity tolerance

Tools

Python

ML frameworks

Job description

Magic in San Francisco is seeking a Member of Technical Staff for the Evals team to design and validate new evaluations and benchmarks across model training and deployment. You will investigate model behavior, measure data quality, and improve evaluation correctness and reproducibility.

Join a team focused on rigorous measurement, reproducibility, and reliability, building tools that make results accessible. Visa sponsorship and relocation support to SF are offered.