AI Model Evaluation Engineer – Build Verifications

Zof AI, Inc.; San Francisco (CA); On-site; USD 140,000 - 190,000; Full time

Benefits offered by this job

MacBook Pro

Premium AI development tools

OpenAI Codex Max

Job summary

Zof AI, Inc. in San Francisco, CA is seeking a Model Evaluation Engineer to build tests that determine whether AI actually works.

This full-time on-site role sits close to the core product, designing eval suites, verification harnesses, and quality gates to turn performance into measurable evidence. The ideal candidate will be skeptical by default, rigorous about measurement, and comfortable turning “it seems to work” into verifiable data.

Qualifications

• Experience testing, evaluating, or QA-ing complex software systems.
• Understanding of how LLM and agent systems fail.
• Strong analytical rigor and skepticism.
• Ability to write code to build harnesses and automation.
• Attention to detail and a high quality bar.
• Clear written and verbal communication.
• Comfort operating in a fast-moving environment.
• High ownership.

Responsibilities

• Design and build eval suites for AI products and agent systems.
• Build verification harnesses that confirm the AI built the right thing.
• Define quality gates that gate what ships and what does not.
• Turn domain expertise and customer requirements into testable checks.
• Hunt failure modes: regressions, hallucinations, and silent errors.
• Make eval results legible to engineers, product, and customers.
• Wire evals into CI and the development loop.
• Raise the standard for what "working" means across the company.

Skills

Testing software systems

LLM/agent failure understanding

Analytical rigor

Automation scripting

Communication

Ownership

Skepticism

Quality assurance

Tools

CI/CD pipelines

Python

Test harnesses

Automation frameworks

Job description

Zof AI, Inc. in San Francisco, CA is seeking a Model Evaluation Engineer to build tests that determine whether AI actually works.

This full-time on-site role sits close to the core product, designing eval suites, verification harnesses, and quality gates to turn performance into measurable evidence. The ideal candidate will be skeptical by default, rigorous about measurement, and comfortable turning “it seems to work” into verifiable data.