Applied AI Benchmarking Engineer

Mercor; San Francisco (CA); On-site; USD 160,000 - 230,000; Full time

Benefits offered by this job

Health benefits

Meal stipend

Relocation support

Housing support

Free Equinox membership

Laundry reimbursement

Wellness reimbursement

Job summary

Mercor is seeking a Research Engineer in San Francisco to own benchmarking pipelines, evaluation systems, and failure-analysis workflows that directly inform how frontier language models are trained and improved.

You will design and run evals, build rubrics and scorers, and collaborate with researchers and data teams to drive model performance. This is an in-person role five days a week in San Francisco.

Qualifications

• Strong applied research background in model evaluation or benchmarking.
• Hands-on coding skills and experience with ML models and evaluation code.
• Solid grasp of data structures, algorithms, and backend systems.

Responsibilities

• Benchmarking: design and maintain benchmarks and metrics for tool use and real-world reasoning.
• Evaluation systems: build end-to-end evaluation runs, scoring, dashboards, and reporting.
• Failure analysis: run systematic analysis on model outputs and feed findings into improvements.
• Rubrics and evaluators: create rubrics and scoring frameworks for training decisions.
• Data quality and usability: quantify data usability and impact on benchmarks.
• Cross-team collaboration: work with researchers and data producers to align evals with objectives.
• Ownership in a fast-paced environment: take ownership of benchmarks, evals, and workflows.

Skills

Applied research

Model evaluation

Benchmarking

Python

ML models

Backend systems

APIs

SQL/NoSQL

Cloud platforms

Data analysis

Job description

Mercor is seeking a Research Engineer in San Francisco to own benchmarking pipelines, evaluation systems, and failure-analysis workflows that directly inform how frontier language models are trained and improved.

You will design and run evals, build rubrics and scorers, and collaborate with researchers and data teams to drive model performance. This is an in-person role five days a week in San Francisco.