LLM Evaluation Engineer - Benchmarks & Failure Analysis

Nous Research; United States; Remote; USD 130,000 - 170,000; Full time

Job summary

Nous Research is seeking an engineer to strengthen our evaluation systems across lab work, benchmark design, and judge calibration. You will ship evaluation infrastructure that researchers rely on from day one, while owning end-to-end eval pipelines and failure analysis.

The role emphasizes high ownership on a small team, with opportunities to shape prompts, environments, and automated graders across GAIA-like benchmarks and new tasks.

Qualifications

• 3+ years in software/ML engineering or similar with concrete evaluation experience.
• Experience with at least one LLM evaluation framework (Harbor, Nemo Evaluator, etc.).
• Hands-on experience with LLMs: prompting, few-shot design, and fine-tuning or RAG.
• Solid Python and clean, tested, version-controlled code.
• Comfort with Git, CI/CD basics, Docker, and the Linux command line.
• Understanding of eval statistics: kappa, confidence intervals, etc.
• Experience with multiple agent benchmarks and adversarial eval is a plus.

Responsibilities

• Run the full eval pipeline end to end and reproduce known results during onboarding.
• Build a judge calibration protocol and document it for re-runs.
• Extend benchmarks with new tasks and accompanying rubric and grader.
• Run failure analysis on model outputs and recommend improvements.
• Own a recurring eval workflow and ship tooling researchers actually use.

Skills

LLM evaluation

Prompting design

Python

Data analysis

Communication

Requirements planning

Tools

Git

CI/CD

Docker

Linux CLI

SSH

tmux

Job description

Nous Research is seeking an engineer to strengthen our evaluation systems across lab work, benchmark design, and judge calibration. You will ship evaluation infrastructure that researchers rely on from day one, while owning end-to-end eval pipelines and failure analysis.

The role emphasizes high ownership on a small team, with opportunities to shape prompts, environments, and automated graders across GAIA-like benchmarks and new tasks.