Evaluation Engineer: AI Coding Benchmarks & Tests

Mercor; United States; Remote; USD 120,000 - 180,000; Full time

Job summary

Mercor is seeking software engineers to build and refine evaluations for frontier models. You’ll transform completed PRs into engineering tasks and use our in‑house framework to benchmark across models such as Claude Code and Codex.

You’ll collaborate with an in‑house research team, write prompts and tests, and help refine grading criteria. Strong programming skills, experience with large codebases, and clear communication are essential to own task development and iteration.

Qualifications

• Proven ability to work with large, complex codebases.
• Experience writing tests and evaluation prompts.
• Excellent communication and independent problem solving.

Responsibilities

• Identify repositories with substantial work to support evaluations.
• Write task prompts and build evaluations using tests and prompts.
• Validate evaluations and refine grading criteria.
• Run evaluations across multiple models and harnesses.
• Document findings and iterate on feedback.

Skills

Programming fundamentals

Codebase navigation

AI tooling experience

Testing skills

Git & CI

Tools

Git

Test runners

CI tooling

Job description

Mercor is seeking software engineers to build and refine evaluations for frontier models. You’ll transform completed PRs into engineering tasks and use our in‑house framework to benchmark across models such as Claude Code and Codex.

You’ll collaborate with an in‑house research team, write prompts and tests, and help refine grading criteria. Strong programming skills, experience with large codebases, and clear communication are essential to own task development and iteration.