Machine Learning Engineer, Evals

Nous Research; United States; Remote; USD 120,000 - 180,000; Full time

Job summary

Nous Research is seeking a highly autonomous engineer to work across the lab on agent capability evaluations, benchmark design, LLM-as-judge systems, and failure analysis. You will own evaluation infrastructure used by researchers from day one, running end-to-end pipelines and documenting calibration processes.

You will extend existing benchmarks, perform failure analyses, and ship recurring eval workflows and dashboards.

Qualifications

• 3+ years in software/ML engineering or research-adjacent roles with evaluation experience.
• Hands-on experience with LLM evaluation frameworks (e.g., Harbor, Nemo Evaluator).
• Hands-on with LLMs: prompting, few-shot design, fine-tuning or RAG; regular use of coding agents.

Responsibilities

• Run full eval pipeline end-to-end and reproduce results during onboarding.
• Build a judge calibration protocol and document it for re-runability.
• Extend benchmarks with new tasks, prompts, environments, rubrics, automated graders and QA.
• Run failure analysis on model outputs and write actionable recommendations.
• Own recurring eval workflow and ship tooling used by researchers.

Skills

Python

LLM evaluation

Calibrations & statistics

Git & CI/CD

Docker

Linux command line

Communication

Experimentation

Prompting & design

Tools

Docker

Git

CI/CD

Linux

Job description

The Role

You\'ll work across the lab on agent capability evals, benchmark design, LLM-as-judge systems, failure analysis, and the infrastructure that ties it together. This is a high-growth, high-ownership role on a small team, and you\'ll ship evaluation infrastructure that researchers depend on from day one.

Responsibilities

• Run the full eval pipeline end to end and reproduce known results during onboarding, pairing with a senior engineer on your first task




• Build a judge calibration protocol: sample human-labeled decisions, measure agreement (κ, per-class P/R), identify drift zones, and document it so anyone can re-run it




• Extend an existing benchmark (GAIA, τ-Bench, SWE-bench slice, etc.) with new tasks targeting known capability gaps, including the prompt, environment, rubric, automated grader, and QA




• Run failure analysis on model outputs: categorize failure modes, quantify prevalence, and write up findings with recommendations for training data, judge prompts, or benchmark changes




• Own a recurring eval workflow (weekly regression suite, judge drift dashboard, red-team evaluation for a new capability) and ship tooling researchers actually use

Qualifications

• 3+ years in software engineering, ML engineering, data science, or a research-adjacent role, with concrete evaluation experience from coursework, an internship, a side project, open source work, or a job




• Experience with at least one LLM evaluation framework (Harbor, Nemo Evaluator, etc.), with real opinions on what it does well and where it falls short




• Hands-on experience with LLMs: prompting, few-shot design, and ideally fine-tuning or RAG; regular use of coding agents




• Solid Python. You write clean, tested, version-controlled code that a colleague could run without you babysitting it




• Comfort with Git, CI/CD basics, Docker, and the Linux command line (SSH, tmux, debugging a remote job)




• Understanding of basic eval statistics: why accuracy misleads on imbalanced judges, what Cohen\'s κ measures, how to think about confidence intervals on a metric




• At least 3 of the following: you can explain why LLM-as-judge needs calibration; you\'ve done failure analysis and can tell model bugs apart from prompt, grader, or retrieval issues; you know at least two agent benchmarks (GAIA, AgentBench, τ-Bench, MINT, SWE-bench, WebShop, ALFWorld) and a limitation of each; you\'ve designed or extended an eval dataset with happy paths, edge cases, and adversarial examples; you\'ve thought about non-determinism in eval, how you sample, how many runs, how you report variance




• You communicate clearly to both researchers and engineers, in the right language for each




• \'You\'re comfortable with ambiguity, can turn a half-formed request into a plan, and know when to ask for help

Preferred

• RLVR / RLHF pipeline experience




• Training data curation experience




• Distributed eval orchestration experience




• Benchmark design from scratch




• Red teaming and adversarial eval experience




• Familiarity with psychometrics or measurement theory