AI Agent Quality Engineer

Cresta

San Francisco (CA)

Remote

CAD 90,000 - 140,000

Full time

7 days ago
Be an early applicant

Get a reply from this employer

Your resume and cover letter, tailored to exactly what they’re hiring for.

Get past ATS filters

Benefits offered by this job

Medical, dental, vision
Flexible PTO
Parental leave
Retirement plan
Remote work setup budget
Wellness stipend
In-office meal program

Job summary

Cresta is building AI-powered customer experience tools. We are seeking an AI Agent Evaluation Engineer to define quality, measure it across conversations, and scale evaluation systems for a production LLM product.

You will design rubrics, run analyses, and collaborate with engineering and product teams to prioritize improvements. This role blends research rigor with hands-on engineering to improve agent reliability across languages and contexts.

Qualifications

  • 3+ years in software/ML/data-intensive research roles.
  • Experience with LLM-based systems in production (1–2 years).
  • Experience building end-to-end evaluation systems for production products.

Responsibilities

  • Review production conversations weekly and identify failure modes.
  • Trace failures to causes and assign owners with evidence.
  • Evaluate multilingual conversations by segment (language, accent, audio quality, intent).
  • Build and operate the evaluation pipeline with automated checks, LLM-based judges, and human review.
  • Calibrate automated judges against human labels and manage regression suites.
  • Design sampling strategies to ensure reported quality changes are real.
  • Document methodology for reproducibility and long-term maintenance.

Skills

Python programming
LLM-based systems in production
Data analysis
Multilingual evaluation
Calibration & quality metrics
Strong written communication

Education

Bachelor's or Master’s degree in Computer Science, Computational Linguistics, Cognitive Science, Statistics, or related field

Tools

LLM APIs
Python tooling
Evaluation tooling

Job description

Cresta is building AI-powered customer experience tools. We are seeking an AI Agent Evaluation Engineer to define quality, measure it across conversations, and scale evaluation systems for a production LLM product.

You will design rubrics, run analyses, and collaborate with engineering and product teams to prioritize improvements. This role blends research rigor with hands-on engineering to improve agent reliability across languages and contexts.

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

AI Agent Evaluation Engineer - Remote
AI Agent Evaluation Engineer - Remote

Cresta • Marion (KS)

Remote
USD 120,000 - 180,000
Remote AI Agent Evaluation Engineer
Remote AI Agent Evaluation Engineer

Cresta • Oregon

Hybrid
USD 120,000 - 180,000
Medical, dental & vision
Flexible PTO
Parental leave
+4
AI Agent Evaluation Engineer
AI Agent Evaluation Engineer

Cresta • New York (NY)

On-site
USD 120,000 - 190,000
AI Agent Evaluation Engineer
AI Agent Evaluation Engineer

Cresta • Marion (KS)

Remote
USD 120,000 - 180,000
AI Agent Evaluation Engineer
AI Agent Evaluation Engineer

Cresta • Oregon

Hybrid
USD 120,000 - 180,000
Medical, dental & vision
Flexible PTO
Parental leave
+4
Lead AI Voice Experience Engineer
Lead AI Voice Experience Engineer

Fdetimes • United States

Remote
USD 205,000 - 270,000
Comprehensive medical, dental, and vis
Flexible PTO
Parental leave
+3
AI Agent Evaluation Engineer
AI Agent Evaluation Engineer

Cresta • San Francisco (CA)

Remote
CAD 90,000 - 140,000
Medical, dental, vision
Flexible PTO
Parental leave
+4
AI Engineer, Agents & Quality - Build Tools & Evaluation
AI Engineer, Agents & Quality - Build Tools & Evaluation

Gold Rose AI • United States

Hybrid
USD 120,000 - 180,000
Lead Engineer - AI Agent Voice Experience
Lead Engineer - AI Agent Voice Experience

Cresta • United States

Remote
USD 150,000 - 210,000
Lead Engineer, Real-Time Voice AI for Agents
Lead Engineer, Real-Time Voice AI for Agents

Devconnectplatform • United States

Remote
USD 205,000 - 270,000
Medical/dental/vision coverage
Flexible PTO
Parental leave
+4