AI Engineer, Post-Training & Evaluation

present of AI; San Francisco (CA); On-site; USD 180,000 - 240,000; Full time

Job summary

Coval is seeking a seasoned ML engineer to design and run evaluation pipelines for scoring voice and chat conversations. You will define metrics, build trusted evaluation systems, and run end-to-end fine-tuning experiments with curated data.

The role rewards clear judgment on what boosts quality and emphasizes calibration against human raters. You will gain hands-on experience with LLM judge design, dataset preparation, and stress testing conversational agents before deployment, aligning with

Qualifications

• Several years of software/ML engineering experience.
• Experience building evaluation systems and trusted pipelines for scoring conversations.
• Ran end-to-end fine-tuning experiments including data preparation and evaluation.

Responsibilities

• Define what a good conversation looks like and establish scoring rubrics.
• Build and operate evaluation systems and measurement tooling for voice and chat models.
• Calibrate scores against human raters and validate stability across releases.
• Prepare data and experiments for LoRA/SFT fine-tuning on open-weight models.

Skills

Evaluation metrics

ML model evaluation

Experimentation

Data pipeline design

Calibration with raters

Education

Tools

LoRA fine-tuning

SFT

Open weight models

Job description

The role centers on building and running the models that score voice and chat conversations, from judge prompts and rubrics to calibration against human raters. A strong candidate has several years of engineering experience, has built evaluation systems they trust, and has run fine tuning experiments end to end including preparing their own data. The work rewards someone who can tell which lever actually improves quality and focus there.

How it fits the AI era

This role is itself about making AI reliable, since the product judges other AI agents before they reach customers. Much of the craft is measurement and judgment: defining what a good conversation looks like and keeping scores honest across releases. The human part is deciding what matters and interpreting results, while the model work scales that judgment across many conversations.

How it builds AI experience

A person here would gain hands on practice with LLM as a judge design, calibration against human raters, and LoRA fine tuning on open weight models. They would also learn to build data pipelines from scored conversations and to track judge reliability over time. That maps directly to the company's focus on stress testing conversational agents before deployment.

Worth showing

• LoRA and SFT
• Eval infrastructure
• Data preparation
• Calibration with raters

This is our own read, written from the employer's posting and the company's Spotlight. The employer's posting is the source of truth for the role.

About Coval

Enterprise simulation and evaluation platform that stress-tests voice and chat AI agents before deployment, per the company.