Senior AI Evaluation Engineer: LLMs & Agent Training

Chegg; United States; Remote; USD 180,000 - 230,000; Full time

Job summary

Chegg is building an AI Evaluation Platform to measure model and agent performance across accuracy, reasoning, safety, and pedagogical quality. You will own evaluation frameworks end-to-end, from framework design through deployment and production monitoring, partnering with ML researchers, applied scientists, and product teams to define what good looks like for models and agents.

This senior role focuses on creating robust evaluation pipelines, rubrics, model-graded evaluators, and

Qualifications

• > 5+ years of software/ML engineering experience in evals, testing, or ML measurement infra.
• > Direct experience with foundation model evaluation and benchmarking in frontier labs or similar.
• > End-to-end ownership of evaluation solutions from design to deployment and monitoring.
• > Experience designing evals for LLMs/AI agents, including benchmark design and human annotation platforms.
• > Proficient in Python; strong in building production-grade data/ML pipelines.
• > Familiar with SOTA training/eval lifecycles (SFT, RLHF/RLAIF/DPO) and post-training evals.
• > Experience with agentic architectures and tooling for tracing, versioning, and experiment tracking.
• > Excellent cross-functional collaboration and critical thinking about metric validity.

Responsibilities

• > Own evaluation solutions end-to-end: design methodology, build pipelines, deploy into training/production, iterate with minimal hand-off.
• > Design and build eval frameworks for LLMs and multi-step agents, including offline benchmarks and online evals.
• > Define rubrics for open-ended tasks, ensure reliable model judgments and bias testing.
• > Develop model-graded evaluators and validate against human judgment.
• > Create agent-specific evaluations: tool use, planning, latency, and cost assessments.
• > Generate golden datasets, adversarial tests, and regression suites to catch regressions.
• > Translate eval results into data curation and training signal improvements (SFT/RLHF/RLAIF).
• > Instrument production data collection for feedback into evaluation/training loops.
• > Build dashboards for researchers/PMs/leadership to monitor model quality trends.

Skills

Python

PyTorch

TensorFlow

production-grade pipelines

statistical judgment

cross-functional collaboration

ML eval design

LLM evaluation

risk and safety thinking

Tools

AWS SageMaker

Bedrock

Databricks MLflow

Unity Catalog

Delta Lake

Job description

Chegg is building an AI Evaluation Platform to measure model and agent performance across accuracy, reasoning, safety, and pedagogical quality. You will own evaluation frameworks end-to-end, from framework design through deployment and production monitoring, partnering with ML researchers, applied scientists, and product teams to define what good looks like for models and agents.

This senior role focuses on creating robust evaluation pipelines, rubrics, model-graded evaluators, and