Staff ML Evaluation & Calibration Engineer

Servicenow; Mountain View (CA); Sur place; USD 180 000 - 260 000; Plein temps

Résumé du poste

Moveworks' AI agents don't just generate text — they act, plan, and change state in enterprise systems. You will own the central problem of scoring agent trajectories across multi-step interactions, turning complex behavior into meaningful judgments that train and improve the system.

This role focuses on designing reliable evaluation signals, combining deterministic validators with probabilistic LLM judgments, and calibrating a process reward model that informs improvements while guarding

Qualifications

• 8+ years in applied ML, data science, or ML‑adjacent engineering with shipped work.
• Ability to turn subjective judgement into measurable signals.
• Strong Python and production-grade software discipline.
• Experience evaluating LLMs and models in real-world settings.
• Comfort with ambiguity and startup pace.

Responsabilités

• Design and calibrate judge rubric systems for agent evaluation.
• Develop deterministic validators and LLM judges for fuzzy judgments.
• Build confidence reporting in scores and workflow integration.
• Collaborate with annotation teams to align human labels.
• Prototype and fine-tune small judge models.
• Explain deviations and maintain robust evaluation pipelines.

Connaissances

Applied ML

Data science

ML engineering

Python

Experimentation

Communication

Outils

Evaluation design

LLM as judge

Prompt engineering

Fine-tuning models

Reward modeling

Description du poste

Moveworks' AI agents don't just generate text — they act, plan, and change state in enterprise systems. You will own the central problem of scoring agent trajectories across multi-step interactions, turning complex behavior into meaningful judgments that train and improve the system.

This role focuses on designing reliable evaluation signals, combining deterministic validators with probabilistic LLM judgments, and calibrating a process reward model that informs improvements while guarding