Senior ML Engineer, Evaluation San Francisco $230,000 – $290,000

Pazau, Inc.; San Francisco (CA); On-site; USD 230,000 - 290,000; Full time

Benefits offered by this job

Equity in Pazau

Parental leave (16 weeks)

401(k) with 4% match

Learning budget $2,000/year

Conference travel

Four weeks PTO + December company week

Job summary

Pazau, Inc. in San Francisco is seeking an experienced ML Evaluation Scientist to join the AI group led by Dr. Mei Tanaka. You will build evaluation suites across languages and industries, and shape how we measure true resolution and policy adherence.

The role emphasizes offline evals, robust statistics, and collaboration across teams; offers a base salary range of $230,000–$290,000 plus equity, with comprehensive benefits and growth opportunities.

Qualifications

• 5+ years in applied ML or equivalent with emphasis on evaluation, measurement or data quality.
• Strong Python programming skills and experience with large-scale data pipelines.
• Rigor in statistics: confidence intervals, inter-rater agreement, bias and benchmarking implications.
• Experience evaluating LLM-based systems in production.

Responsibilities

• Build and maintain evaluation suites across 40+ industries and 30 languages.
• Develop LLM-as-judge graders and calibrate against expert raters.
• Own the release gate: offline evals and simulations before product rollout.
• Measure true resolution, policy adherence, and reduce hallucinated actions or escalations.
• Co-author the 2027 Pazau resolution benchmark.

Skills

Python

Large-scale data pipelines

Experiment tracking

LLM evaluation

Job description

About the role

Our headline number is resolution rate. Pazau Agent resolves 67% of conversations end-to-end on average. That number is only worth something if we can measure it honestly. The Evaluation team, part of Dr. Mei Tanaka's AI group, builds the systems that decide whether a new model, prompt or Procedure change ships. It also produces the annual resolution benchmark we publish.

The base salary range for this role is $230,000 – $290,000. Final offers depend on level and location, and every offer includes equity.

What you’ll do

• Build and maintain evaluation suites across 40+ industries and 30 languages, drawn from consented, redacted conversations
• Develop LLM-as-judge graders and calibrate them against expert human raters, with published agreement rates
• Own the release gate: every model or runtime change runs through offline evals and simulation before it reaches a customer
• Measure what customers care about: true resolution, policy adherence, hallucinated Actions and unnecessary escalations
• Co-author the 2027 Pazau resolution benchmark

What you’ll bring

• 5+ years in applied ML, with significant time on evaluation, measurement or data quality
• Strong Python, plus experience with large-scale data pipelines and experiment tracking
• Rigor about statistics: confidence intervals, inter-rater agreement and the ways a benchmark can mislead
• Experience evaluating LLM-based systems in production

Nice to have

• Published work on evaluation methodology
• Experience with multilingual or speech evaluation
• Background in a regulated domain such as healthcare or finance
• Medical, dental and vision for you and your dependents, with 100% of premiums covered
• Equity in Pazau, with a 10-year exercise window after you leave
• 16 weeks of fully paid parental leave for every parent
• 401(k) with a 4% match
• $2,000 a year for learning, plus conference travel when you’re speaking
• Four weeks of paid time off, plus a company-wide week off at the end of December
• Recruiter call (30 min). We walk through the role, compensation and timeline, and answer your questions first.
• Hiring manager conversation (45 min). A conversation about work you’re proud of and the problems this team owns.
• Research deep dive (90 min). You’ll present a piece of past evaluation work and we’ll dig into the choices you made. Then we’ll look at a flawed eval together.
• Final interviews (3–4 hours, virtual or on-site). Three or four conversations with the people you’d work with every day, including one cross-functional partner.
• Founder conversation and references. A 30-minute conversation with Amara or Lukas, then two references you choose.

Most candidates go from first call to offer in three to four weeks. You’ll hear back from us within three business days after every stage, and every candidate who reaches final interviews gets written feedback.

Sound like your next job?

We reply to every application within five business days.