AI Evaluation Engineer, Benchmarks & Tooling

Bake AI Inc.; San Mateo (CA); On-site; USD 180,000 - 240,000; Full time

Job summary

Bake AI Inc. is hiring a Member of Technical Staff in San Mateo, California to build evaluations grounded in real-world tasks and workflows. You will design tasks, environments, and verification methods that reflect how models perform in practice, including tool use, multi-step reasoning, and handling changing state.

Work with researchers and engineers to turn requirements into reproducible evaluation pipelines, analyze results, and drive meaningful improvements in model reliability and safety.

Qualifications

• Evidence of independently building a substantial evaluation, benchmark, research tool, or ML system.
• Strong Python skills and practical software engineering experience.
• Hands-on experience evaluating language models or agents, including task design, grading, error analysis.
• Solid experimental and statistical judgment: sampling, controlled comparisons, uncertainty.
• Ability to turn an ambiguous workflow into concrete tasks and testable outcomes.
• Careful technical writing and collaboration with researchers and engineers.

Responsibilities

• Design evaluations for realistic work with multi-step tasks and external tools.
• Build task environments, datasets, and reference solutions with researchers.
• Develop executable verifiers and grading systems with clear rubrics.
• Create reproducible evaluation pipelines on sandbox infrastructure with traceability.
• Analyze model trajectories to identify failure modes in planning and tool use.
• Evaluate reliability, quality, latency, and cost alongside scores.
• Turn observed failures into regression suites and guard against data leakage.
• Collaborate with partner teams to communicate limitations and track progress.

Skills

Python

Software engineering

Experimentation

Job description

Bake AI Inc. is hiring a Member of Technical Staff in San Mateo, California to build evaluations grounded in real-world tasks and workflows. You will design tasks, environments, and verification methods that reflect how models perform in practice, including tool use, multi-step reasoning, and handling changing state.

Work with researchers and engineers to turn requirements into reproducible evaluation pipelines, analyze results, and drive meaningful improvements in model reliability and safety.