Remote AI Agent Evaluation Engineer

Cresta

Oregon

Hybrid

USD 120,000 - 180,000

Full time

7 days ago
Be an early applicant

A complete application in a minute

Tailored resume and cover letter, ready to send.

Get past ATS filters

Benefits offered by this job

Medical, dental & vision
Flexible PTO
Parental leave
Retirement plan
Remote work setup budget
Wellness stipend
In-office meal program

Job summary

Cresta in Oregon is seeking an AI Agent Evaluation Engineer to define quality, build scalable evaluation systems, and measure performance of our AI receptionist across conversations. You will annotate data, design metrics, calibrate judges, and collaborate with engineers, product, and language teams to improve reliability and user experience.

This role blends hands-on analysis with engineering execution in a fast-moving AI company, with opportunities to influence product direction and customer

Qualifications

  • 3+ years in software engineering, applied ML, or data-intensive research with 1–2 years hands-on with LLMs in production.
  • Bachelor’s or Master’s in CS, Computational Linguistics, Cognitive Science, Statistics, or equivalent practical experience.
  • Built and operated an end-to-end evaluation system for a production LLM product: rubric, labels, automated judges, regression suite.
  • Deep understanding of evaluation methodology: golden sets, judges, regression suites, and calibration.
  • Fluent in at least one language other than English; reading a trace to understand agent actions.
  • Strong Python programming skills; able to build evaluation tooling independently; fine-tuning not required.
  • Willing to do non-scalable work weekly: label conversations, build a failure taxonomy from scratch.
  • Strong written communication; author the guidelines others follow.

Responsibilities

  • Define and maintain the evaluation rubric across adherence, outcome, and conversation experience.
  • Write the annotation guidelines and failure taxonomy; run calibration sessions and measure inter-rater agreement.
  • Classify and prioritize failure modes by severity and frequency.
  • Review production conversations weekly; cluster failures for automated evaluations.
  • Trace each failure to its likely cause and route to the right owner with evidence.
  • Evaluate across languages and report quality by segment (language, accent, audio quality, intent).
  • Build and operate the evaluation pipeline that scores every production conversation.
  • Calibrate automated judges against human labels; treat judge error as a risk.
  • Design sampling strategies to ensure reported quality changes are real.
  • Build and maintain the golden set and regression suite for validation.

Skills

Python
LLM experience
Data analysis
Communication
Prompting

Education

BSc/MSc in CS/Computational Linguistics/Statistics

Tools

Python
LLM APIs

Job description

Cresta in Oregon is seeking an AI Agent Evaluation Engineer to define quality, build scalable evaluation systems, and measure performance of our AI receptionist across conversations. You will annotate data, design metrics, calibrate judges, and collaborate with engineers, product, and language teams to improve reliability and user experience.

This role blends hands-on analysis with engineering execution in a fast-moving AI company, with opportunities to influence product direction and customer

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

AI Agent Evaluation Engineer - Remote
AI Agent Evaluation Engineer - Remote

Cresta • Marion (KS)

Remote
USD 120,000 - 180,000
AI Agent Quality Engineer
AI Agent Quality Engineer

Cresta • San Francisco (CA)

Remote
CAD 90,000 - 140,000
Medical, dental, vision
Flexible PTO
Parental leave
+4
AI Agent Evaluation Engineer
AI Agent Evaluation Engineer

Cresta • New York (NY)

On-site
USD 120,000 - 190,000
AI Agent Evaluation Engineer
AI Agent Evaluation Engineer

Cresta • Marion (KS)

Remote
USD 120,000 - 180,000
AI Agent Evaluation Engineer
AI Agent Evaluation Engineer

Cresta • Oregon

Hybrid
USD 120,000 - 180,000
Medical, dental & vision
Flexible PTO
Parental leave
+4
AI Agent Evaluation Engineer
AI Agent Evaluation Engineer

Cresta • San Francisco (CA)

Remote
CAD 90,000 - 140,000
Medical, dental, vision
Flexible PTO
Parental leave
+4
Lead Engineer - AI Agent Voice Experience
Lead Engineer - AI Agent Voice Experience

Cresta • United States

Remote
USD 150,000 - 210,000
Lead AI Voice Engineer – Real-Time Conversation Systems
Lead AI Voice Engineer – Real-Time Conversation Systems

Cresta • United States

Remote
USD 150,000 - 210,000
Senior Forward Deployed Engineer (AI Agent) - UK
Senior Forward Deployed Engineer (AI Agent) - UK

Cresta • United States

Remote
USD 94,000 - 147,000
Lead AI Voice Experience Engineer
Lead AI Voice Experience Engineer

Fdetimes • United States

Remote
USD 205,000 - 270,000
Comprehensive medical, dental, and vis
Flexible PTO
Parental leave
+3