LLM Evaluation Engineer: Benchmark & Infra Lead

Nous Research; United States; A distancia; USD 120.000 - 180.000; Jornada completa

Descripción de la vacante

Nous Research is seeking a highly autonomous engineer to work across the lab on agent capability evaluations, benchmark design, LLM-as-judge systems, and failure analysis. You will own evaluation infrastructure used by researchers from day one, running end-to-end pipelines and documenting calibration processes.

You will extend existing benchmarks, perform failure analyses, and ship recurring eval workflows and dashboards.

Formación

• 3+ years in software/ML engineering or research-adjacent roles with evaluation experience.
• Hands-on experience with LLM evaluation frameworks (e.g., Harbor, Nemo Evaluator).
• Hands-on with LLMs: prompting, few-shot design, fine-tuning or RAG; regular use of coding agents.

Responsabilidades

• Run full eval pipeline end-to-end and reproduce results during onboarding.
• Build a judge calibration protocol and document it for re-runability.
• Extend benchmarks with new tasks, prompts, environments, rubrics, automated graders and QA.
• Run failure analysis on model outputs and write actionable recommendations.
• Own recurring eval workflow and ship tooling used by researchers.

Conocimientos

Python

LLM evaluation

Calibrations & statistics

Git & CI/CD

Docker

Linux command line

Communication

Experimentation

Prompting & design

Herramientas

Docker

Git

CI/CD

Linux

Descripción del empleo

Nous Research is seeking a highly autonomous engineer to work across the lab on agent capability evaluations, benchmark design, LLM-as-judge systems, and failure analysis. You will own evaluation infrastructure used by researchers from day one, running end-to-end pipelines and documenting calibration processes.

You will extend existing benchmarks, perform failure analyses, and ship recurring eval workflows and dashboards.