Member of Technical Staff, Evaluations

Bake AI Inc.; San Mateo (CA); On-site; USD 180,000 - 240,000; Full time

Job summary

Bake AI Inc. is hiring a Member of Technical Staff in San Mateo, California to build evaluations grounded in real-world tasks and workflows. You will design tasks, environments, and verification methods that reflect how models perform in practice, including tool use, multi-step reasoning, and handling changing state.

Work with researchers and engineers to turn requirements into reproducible evaluation pipelines, analyze results, and drive meaningful improvements in model reliability and safety.

Qualifications

• Evidence of independently building a substantial evaluation, benchmark, research tool, or ML system.
• Strong Python skills and practical software engineering experience.
• Hands-on experience evaluating language models or agents, including task design, grading, error analysis.
• Solid experimental and statistical judgment: sampling, controlled comparisons, uncertainty.
• Ability to turn an ambiguous workflow into concrete tasks and testable outcomes.
• Careful technical writing and collaboration with researchers and engineers.

Responsibilities

• Design evaluations for realistic work with multi-step tasks and external tools.
• Build task environments, datasets, and reference solutions with researchers.
• Develop executable verifiers and grading systems with clear rubrics.
• Create reproducible evaluation pipelines on sandbox infrastructure with traceability.
• Analyze model trajectories to identify failure modes in planning and tool use.
• Evaluate reliability, quality, latency, and cost alongside scores.
• Turn observed failures into regression suites and guard against data leakage.
• Collaborate with partner teams to communicate limitations and track progress.

Skills

Python

Software engineering

Experimentation

Job description

Bake AI builds environments, evaluations, and sandbox infrastructure for self-evolving AI. We are hiring one Member of Technical Staff to develop evaluations grounded in real-world tasks and workflows. This is a full-time position based in San Mateo, California.

You will turn open-ended questions about model capability and reliability into evaluations that teams can trust. The work starts with understanding what successful performance means in practice: completing a task, using tools correctly, handling incomplete information, and producing an outcome that meets the user's needs. You will work with researchers, engineers, and domain experts to translate those requirements into representative tasks and defensible measures of success.

You will own the path from task design and verification through execution, analysis, and ongoing use. That includes building the evaluation software, inspecting model behavior, and investigating whether a score reflects a meaningful improvement. We are looking for someone who combines strong engineering with careful experimental judgment and can carry an ambiguous problem through to a useful, reproducible result.

What you’ll do

• Design evaluations for realistic work, including tasks that require multiple steps, external tools, and interaction with changing state. Define success criteria that reflect the quality and completeness of the final outcome.
• Build task environments, datasets, and reference solutions with researchers and domain experts. Establish appropriate difficulty, coverage, and held-out cases, and document what an evaluation can and cannot tell us.
• Develop executable verifiers and grading systems. Where human or model judgment is needed, create clear rubrics, calibrate against expert review, and investigate disagreement, bias, and scoring shortcuts.
• Build reproducible evaluation pipelines on sandbox infrastructure, with controlled configurations, explicit resource budgets, and traceable inputs, tool calls, outputs, and artifacts.
• Analyze model and agent trajectories to identify failure modes in planning, tool use, reasoning, and recovery. Distinguish model limitations from ambiguous tasks, environment failures, and errors in the evaluator.
• Design controlled comparisons and repeated trials, quantify uncertainty, and examine performance across task types. Evaluate reliability, quality, latency, and cost alongside aggregate scores.
• Turn observed failures into targeted evaluations and regression suites. Audit for data leakage, reward hacking, and brittle scoring so that improvements remain meaningful as models and workflows change.
• Work with partner teams to make evaluation findings actionable, communicate limitations clearly, and maintain the tools and datasets used to track progress over time.

What you’ll bring

• Evidence of independently building a substantial evaluation, benchmark, research tool, or ML system. You can explain the question it addressed, how you established validity, and the decisions its results supported.
• Strong Python skills and practical software engineering experience. You can build reliable execution and data-processing pipelines, integrate model and tool APIs, and debug unfamiliar systems.
• Hands-on experience evaluating language models or agents, including task design, grading, error analysis, and interpretation of results beyond a single headline metric.
• Solid experimental and statistical judgment: representative sampling, controlled comparisons, repeated measurements, uncertainty, and awareness of confounding factors.
• The ability to turn an ambiguous workflow into concrete tasks and testable outcomes, while recognizing where automated checks are insufficient and expert judgment is necessary.
• Careful technical writing and the ability to work closely with researchers, engineers, and domain experts. You can set a technical direction, challenge assumptions, and deliver with limited oversight.

Preferred experience

• Developing evaluations used by a research team, a production product, or an external benchmark community.
• Building executable environments for coding, research, tool use, or other professional workflows with verifiable outcomes.
• Designing human evaluation studies, annotation guidelines, model-based graders, or methods for measuring agreement and evaluator reliability.
• Investigating reward hacking, benchmark contamination, distribution shift, or failures that only emerge over extended task execution.
• Using evaluation results to guide post-training, data collection, model selection, or product improvements.
• Research publications or open-source contributions in evaluation, agent systems, or related areas, with substantial responsibility for implementation and experiments.