AI Evaluation Engineer - Build Eval Suites & Quality Gates

Zof AI; San Francisco, Northern (CA, KY); Hybrid; USD 140,000 - 190,000; Full time

Job summary

Zof AI in San Francisco is seeking a Model Evaluation Engineer to design eval suites, verification harnesses, and quality gates for AI systems. You will translate customer needs into testable checks and drive measurable evidence that the product behaves as intended.

Focus on identifying failure modes, such as regressions and hallucinations, and wiring evaluations into CI to raise the standard of what "working" means across the company.

Qualifications

• Experience testing, evaluating, or QA-ing complex software systems.
• Understanding of how LLM and agent systems fail.
• Strong analytical rigor and skepticism.
• Ability to write code to build harnesses and automation.
• Attention to detail and a high quality bar.
• Clear written and verbal communication.
• Comfort operating in a fast-moving environment.
• High ownership.

Responsibilities

• Design and build eval suites for AI products and agent systems.
• Build verification harnesses that confirm the AI built the right thing.
• Define quality gates that gate what ships and what does not.
• Turn domain expertise and customer requirements into testable checks.
• Hunt failure modes: regressions, hallucinations, and silent errors.
• Make eval results legible to engineers, product, and customers.
• Wire evals into CI and the development loop.
• Raise the standard for what "working" means across the company.

Skills

QA testing

LLM failures

Analytical rigor

Test automation

Attention to detail

Clear communication

Fast-paced environment

Ownership

Job description

Zof AI in San Francisco is seeking a Model Evaluation Engineer to design eval suites, verification harnesses, and quality gates for AI systems. You will translate customer needs into testable checks and drive measurable evidence that the product behaves as intended.

Focus on identifying failure modes, such as regressions and hallucinations, and wiring evaluations into CI to raise the standard of what "working" means across the company.