AI Evaluation Engineer: LLM Judging & Calibration

present of AI; San Francisco (CA); On-site; USD 180,000 - 240,000; Full time

Job summary

Coval is seeking a seasoned ML engineer to design and run evaluation pipelines for scoring voice and chat conversations. You will define metrics, build trusted evaluation systems, and run end-to-end fine-tuning experiments with curated data.

The role rewards clear judgment on what boosts quality and emphasizes calibration against human raters. You will gain hands-on experience with LLM judge design, dataset preparation, and stress testing conversational agents before deployment, aligning with

Qualifications

• Several years of software/ML engineering experience.
• Experience building evaluation systems and trusted pipelines for scoring conversations.
• Ran end-to-end fine-tuning experiments including data preparation and evaluation.

Responsibilities

• Define what a good conversation looks like and establish scoring rubrics.
• Build and operate evaluation systems and measurement tooling for voice and chat models.
• Calibrate scores against human raters and validate stability across releases.
• Prepare data and experiments for LoRA/SFT fine-tuning on open-weight models.

Skills

Evaluation metrics

ML model evaluation

Experimentation

Data pipeline design

Calibration with raters

Education

Tools

LoRA fine-tuning

SFT

Open weight models

Job description

Coval is seeking a seasoned ML engineer to design and run evaluation pipelines for scoring voice and chat conversations. You will define metrics, build trusted evaluation systems, and run end-to-end fine-tuning experiments with curated data.

The role rewards clear judgment on what boosts quality and emphasizes calibration against human raters. You will gain hands-on experience with LLM judge design, dataset preparation, and stress testing conversational agents before deployment, aligning with