Senior Model Evaluation Engineer – LLM Benchmarking

cohere; New York (NY); Hybrid; USD 150,000 - 210,000; Full time

Benefits offered by this job

A weekly lunch stipend of $75/£75 or?e

Full health and dental benefits

RRSP matching, 401K, Pension Scheme

100% Parental Leave top‑up

Annual enrichment benefits and cowork/

Job summary

Cohere is seeking a Senior Research Engineer, Model Evaluation, to create next‑generation evaluation methods and scalable infrastructure. You will develop benchmarks, datasets, and environments to measure frontier model capabilities, and you will push the state‑of‑the‑art in LLM evaluation methods while building tools used across technical staff and leadership.

You will collaborate with leading researchers and engineers, focusing on improving evaluation efficiency and scalable dataset

Qualifications

• You enjoy pushing the limits of LLM evaluation and have built high‑quality evaluation resources (datasets, simulators, environments).
• You have a track record of developing new methods or data to evaluate LLMs, e.g. publications or benchmarks.
• You have deep experience building with and around LLMs and you’ve built tools to analyze performance.

Responsibilities

• Develop evaluation benchmarks, datasets, and environments for measuring cutting‑edge model capabilities.
• Conduct research to advance LLM evaluation methods, including training LLM judges and efficient evaluation.
• Build scalable tools to investigate and understand evaluation results for the org and leadership.
• Collaborate with top researchers and engineers to push state‑of‑the‑art in evaluation.

Skills

LLM evaluation

Research engineering

Software engineering

Publications / benchmarks

Tools

Python

PyTorch / TensorFlow

Datasets & evaluation tooling

Job description

Cohere is seeking a Senior Research Engineer, Model Evaluation, to create next‑generation evaluation methods and scalable infrastructure. You will develop benchmarks, datasets, and environments to measure frontier model capabilities, and you will push the state‑of‑the‑art in LLM evaluation methods while building tools used across technical staff and leadership.

You will collaborate with leading researchers and engineers, focusing on improving evaluation efficiency and scalable dataset