AI Engineer, Post-Training & Evaluation

Coval; San Francisco (CA); On-site; USD 180,000 - 240,000; Full time

Job summary

Coval is the simulation and evaluation platform for voice AI. We help companies answer the question most can't: do their voice agents actually work? Not in a demo.

At scale, in production, with real users. Own the model side end to end. You’ll design and ship the LLM-as-a-judge evaluations that score conversations at scale, then sharpen them with LoRA fine-tunes on open-weight models and non-tuning work like tool use and context engineering.

Qualifications

• 4+ years as an engineer with strong systems instincts.
• Track record of shipping measurable quality gains by rigorous measurement.
• Experience building eval infrastructure you trust.
• Experience with LoRA/SFT experiments end to end and data preparation.

Responsibilities

• Design and ship the LLM judges that score conversations: rubrics, judge prompting, and calibration against human raters.
• Build the reliability tracking that keeps judges honest from release to release.
• Run post-training where training beats prompting: LoRA adapters on open-weight models.
• Do non-tuning work: tool use, structured outputs, and context engineering.
• Identify which lever actually moves the metric and drive it forward.

Skills

LLM evaluation

LoRA fine-tuning

Evaluation infrastructure

Context engineering

Tool use

Tools

LoRA adapters

Job description

Simulation & Evaluation that scales voice and chat AI agents

Own the models behind Coval’s judges. Coval is the evaluation platform for conversational AI: we help enterprise teams simulate, monitor and improve voice and chat agents before those agents ever reach a customer. Because our product is the judge, this role sits at the center of the company.

You’ll own the model side end to end. You’ll design and ship the LLM-as-a-judge evaluations that score voice and chat conversations at scale, then sharpen them with LoRA fine-tunes on open-weight models and non-tuning work like tool use and context engineering. Your improvements ship directly into the product enterprises rely on to launch dependable voice agents.

What you’ll do

• Design and ship the LLM judges that score conversations: rubrics, judge prompting, and calibration against human raters
• Build the reliability tracking that keeps judges honest from release to release
• Run post-training where training beats prompting: LoRA adapters on open-weight models, trained on data distilled from our own scored conversations
• Everywhere else, do the non-tuning capability work: tool use, structured outputs, and context engineering
• Keep asking which lever actually moves the metric, and focus there

What we’re looking for

• 4+ years as an engineer, with strong systems instincts
• A track record of shipping measurable quality gains by pairing rigorous measurement with targeted training, not vibes and prompt tweaks
• You’ve built eval infrastructure you actually trust
• You’ve run LoRA/SFT experiments end to end, including preparing your own data rather than relying on clean data from other teams, so you know which lever really wins
• High agency: on a small Series A team, nobody hands you tickets. You figure out what matters next and drive it there
• Ambitious and kind. You’ll be part of the group that founds our engineering org, sets its bar and grows the team

Coval is the simulation and evaluation platform for voice AI. We help companies answer the question most can't: do their voice agents actually work? Not in a demo. At scale, in production, with real users.

Most teams building voice agents are flying blind. They ship a demo that works, deploy it, and discover weeks later that 40% of conversations are failing. No evaluation infrastructure. No way to catch regressions before users do.

We built Coval because we've seen this movie before. Brooke led evaluation infrastructure at Waymo, where it takes millions of simulated miles before a vehicle touches a public road. Voice agents need the same rigor. Right now, almost nobody has it.

Nine people, backed by YC, closing six-figure enterprise deals with Fortune 500 companies, growing revenue 10x year over year. The space is moving fast. We're at the center of it.

What It's Like Here

We work in-person in SoMa, San Francisco. The office is shoes-off, full of plants, and dog-friendly. We bike, hike, skate, and run half-marathons together.

The team comes from Waymo, Zoox, Apple, and Google. We're hard on our work but never hard on each other. We ship on Sundays because we want to, not because someone told us to. We move at AI speed: what used to take weeks happens in hours.

Our operating principle is "wholesome and unhinged." We are relentless about making things happen, but we don't take ourselves too seriously. If you thrive in ambiguity, move fast, and want to be one of the first people building the sales foundations at a company that's already closing Fortune 500 deals, this is it.

Voice AI is where the market is going. We're already there. This is the best time to join.