AI Agent Evaluation Engineer - Remote

Cresta

Marion (KS)

Remote

USD 120,000 - 180,000

Full time

7 days ago
Be an early applicant

Get a reply from this employer

Your resume and cover letter, tailored to exactly what they’re hiring for.

Get past ATS filters

Job summary

Cresta is hiring an AI Agent Evaluation Engineer to shape how we measure agent quality across conversations. You will design evaluation rubrics, build automated judges, and calibrate them against human labels to ensure reliable metrics.

You’ll work on multilingual evaluations and collaborate with ML, engineering, and product teams to improve agent performance across languages and channels. You will develop scalable evaluation pipelines, define failure taxonomies, and communicate results clearly

Qualifications

  • 3+ years in software engineering, ML, or data-intensive research roles.
  • 1–2 years hands-on with LLM-based systems in production.
  • Bachelor's or Master's in CS, Computational Linguistics, Cognitive Science, Statistics, or related field, or equivalent experience.
  • Built and operated an end-to-end evaluation system for a production LLM product: rubric, human labels, automated judge, regression suite.
  • Fluent understanding of prompts, tool calls, context handling, multi-turn state in LLM agents.
  • Strong programming in Python and experience with LLM APIs; able to build evaluation tooling independently.
  • Ability to write a rubric that reviewers score consistently.

Responsibilities

  • Define and maintain evaluation rubric across adherence, outcome, and experience.
  • Write annotation guidelines and failure taxonomy; run calibration sessions.
  • Review production conversations weekly; cluster failures and turn into automations.
  • Trace failures to causes (prompt, knowledge gap, tool bug, ASR) and assign owners.
  • Evaluate across languages and report quality by segment (language, accent, audio quality, intent).
  • Build and operate the evaluation pipeline with automated checks, LLM-based judges, and human review.
  • Calibrate automated judges against human labels; manage calibration risk.
  • Design sampling strategies to ensure reported quality changes are real.

Skills

Python programming
LLM APIs
Software engineering
Applied ML
Data analysis
Multilingual evaluation

Education

Bachelor's or Master's in CS or related field
Equivalent practical experience

Tools

Python
LLM tooling

Job description

Cresta is hiring an AI Agent Evaluation Engineer to shape how we measure agent quality across conversations. You will design evaluation rubrics, build automated judges, and calibrate them against human labels to ensure reliable metrics.

You’ll work on multilingual evaluations and collaborate with ML, engineering, and product teams to improve agent performance across languages and channels. You will develop scalable evaluation pipelines, define failure taxonomies, and communicate results clearly

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

AI Agent Quality Engineer
AI Agent Quality Engineer

Cresta • San Francisco (CA)

Remote
CAD 90,000 - 140,000
Medical, dental, vision
Flexible PTO
Parental leave
+4
Remote AI Agent Evaluation Engineer
Remote AI Agent Evaluation Engineer

Cresta • Oregon

Hybrid
USD 120,000 - 180,000
Medical, dental & vision
Flexible PTO
Parental leave
+4
AI Agent Evaluation Engineer
AI Agent Evaluation Engineer

Cresta • New York (NY)

On-site
USD 120,000 - 190,000
AI Agent Evaluation Engineer
AI Agent Evaluation Engineer

Cresta • Marion (KS)

Remote
USD 120,000 - 180,000
AI Agent Evaluation Engineer
AI Agent Evaluation Engineer

Cresta • Oregon

Hybrid
USD 120,000 - 180,000
Medical, dental & vision
Flexible PTO
Parental leave
+4
AI Agent Evaluation Engineer
AI Agent Evaluation Engineer

Cresta • San Francisco (CA)

Remote
CAD 90,000 - 140,000
Medical, dental, vision
Flexible PTO
Parental leave
+4
Remote AI Growth Engineer: Build AI Agents & Web Platform
Remote AI Growth Engineer: Build AI Agents & Web Platform

Cresta • Myrtle Point (OR)

Remote
USD 160,000 - 280,000
Comprehensive health benefits
Remote work setup budget
Paid parental leave
+3
AI Agent Evaluator — Remote Rubric & Feedback Specialist
AI Agent Evaluator — Remote Rubric & Feedback Specialist

Auraone • United States

Remote
USD 34,000 - 55,000
Senior AI Agent Engineer — Deploy & Integrate AI Agents
Senior AI Agent Engineer — Deploy & Integrate AI Agents

Cresta • United States

Remote
USD 94,000 - 147,000
Lead Engineer - AI Agent Voice Experience
Lead Engineer - AI Agent Voice Experience

Cresta • United States

Remote
USD 150,000 - 210,000