Staff Engineer, Enterprise Evaluation Platform

Doist; San Francisco (CA); On-site; USD 150,000 - 190,000; Full time

Benefits offered by this job

Relocation bonus

Housing stipend

Meals stipend

Equity grant

Free Equinox membership

Laundry reimbursement

Wellness reimbursement

Health insurance

Job summary

Mercor in San Francisco is seeking an experienced Platform Engineer for our Enterprise Agent Eval Systems. You will design verifiers, build offline environments, and craft scalable grading infrastructure that works across customers, domains, and tasks.

You will collaborate with the Enterprise Platform and Applied AI teams to codify evaluation practices, reduce system friction, and enable reliable, measurable improvements in agent performance at scale.

Qualifications

• Experience in building evaluation systems for agent or LLM runtimes and benchmarks.
• Ability to turn qualitative quality notions into measurable rubrics for model improvements.
• Strong software engineering fundamentals and independent problem solving.

Responsibilities

• Define golden sets by decomposing real tasks and encoding expert quality bars.
• Build verifiers over agent trajectories and outputs, calibrated and hard to game.
• Build the evaluation platform that runs offline environments and grading at scale.
• Analyze production trajectories and turn failure modes into regression tests.
• Run the optimization loop across models, prompts, skills, and harnesses.
• Own rollout gates that decide whether an agent change ships.
• Collaborate with the Enterprise Platform team and Applied AI engineers embedded with customers.

Skills

Agent engineering

Evaluation suites

LLM/agent systems

Judgment design

Tools

Harbor environments

RL environments

Job description

Mercor in San Francisco is seeking an experienced Platform Engineer for our Enterprise Agent Eval Systems. You will design verifiers, build offline environments, and craft scalable grading infrastructure that works across customers, domains, and tasks.

You will collaborate with the Enterprise Platform and Applied AI teams to codify evaluation practices, reduce system friction, and enable reliable, measurable improvements in agent performance at scale.