AI Evaluation Engineer — Voice & Chat at Scale

Coval; San Francisco (CA); On-site; USD 180,000 - 240,000; Full time

Job summary

Coval is the simulation and evaluation platform for voice AI. We help companies answer the question most can't: do their voice agents actually work? Not in a demo.

At scale, in production, with real users. Own the model side end to end. You’ll design and ship the LLM-as-a-judge evaluations that score conversations at scale, then sharpen them with LoRA fine-tunes on open-weight models and non-tuning work like tool use and context engineering.

Qualifications

• 4+ years as an engineer with strong systems instincts.
• Track record of shipping measurable quality gains by rigorous measurement.
• Experience building eval infrastructure you trust.
• Experience with LoRA/SFT experiments end to end and data preparation.

Responsibilities

• Design and ship the LLM judges that score conversations: rubrics, judge prompting, and calibration against human raters.
• Build the reliability tracking that keeps judges honest from release to release.
• Run post-training where training beats prompting: LoRA adapters on open-weight models.
• Do non-tuning work: tool use, structured outputs, and context engineering.
• Identify which lever actually moves the metric and drive it forward.

Skills

LLM evaluation

LoRA fine-tuning

Evaluation infrastructure

Context engineering

Tool use

Tools

LoRA adapters

Job description

Coval is the simulation and evaluation platform for voice AI. We help companies answer the question most can't: do their voice agents actually work? Not in a demo.

At scale, in production, with real users. Own the model side end to end. You’ll design and ship the LLM-as-a-judge evaluations that score conversations at scale, then sharpen them with LoRA fine-tunes on open-weight models and non-tuning work like tool use and context engineering.