Senior ML Engineer — LLM Evaluation & Benchmarking

Pazau, Inc.; San Francisco (CA); On-site; USD 230,000 - 290,000; Full time

Benefits offered by this job

Equity in Pazau

Parental leave (16 weeks)

401(k) with 4% match

Learning budget $2,000/year

Conference travel

Four weeks PTO + December company week

Job summary

Pazau, Inc. in San Francisco is seeking an experienced ML Evaluation Scientist to join the AI group led by Dr. Mei Tanaka. You will build evaluation suites across languages and industries, and shape how we measure true resolution and policy adherence.

The role emphasizes offline evals, robust statistics, and collaboration across teams; offers a base salary range of $230,000–$290,000 plus equity, with comprehensive benefits and growth opportunities.

Qualifications

• 5+ years in applied ML or equivalent with emphasis on evaluation, measurement or data quality.
• Strong Python programming skills and experience with large-scale data pipelines.
• Rigor in statistics: confidence intervals, inter-rater agreement, bias and benchmarking implications.
• Experience evaluating LLM-based systems in production.

Responsibilities

• Build and maintain evaluation suites across 40+ industries and 30 languages.
• Develop LLM-as-judge graders and calibrate against expert raters.
• Own the release gate: offline evals and simulations before product rollout.
• Measure true resolution, policy adherence, and reduce hallucinated actions or escalations.
• Co-author the 2027 Pazau resolution benchmark.

Skills

Python

Large-scale data pipelines

Experiment tracking

LLM evaluation

Job description

Pazau, Inc. in San Francisco is seeking an experienced ML Evaluation Scientist to join the AI group led by Dr. Mei Tanaka. You will build evaluation suites across languages and industries, and shape how we measure true resolution and policy adherence.

The role emphasizes offline evals, robust statistics, and collaboration across teams; offers a base salary range of $230,000–$290,000 plus equity, with comprehensive benefits and growth opportunities.