Founding AI Evaluation Engineer

US Health Partners, LLC; San Francisco (CA); On-site; USD 120,000 - 180,000; Full time

Benefits offered by this job

Founding equity

Competitive cash

Office in FiDi

Meals provided

Healthcare coverage

Equinox membership or fitness stipend

AI tools budget

Relocation support

Job summary

Litmus is building evaluation tools to measure AI agents performing real work. You will design metrics for domains without ground truth and implement verifiers, graders, and simulation environments that reflect real-world tasks. You will ensure numbers are trustworthy and scalable to customers.

You will ship hands-on evaluation systems, shaping how agents are judged across industries and turning insights into product capabilities. Relocation to San Francisco is supported.

Qualifications

• Experience building evals, benchmarks, verifiers, or RL environments for LLMs or agents.
• Ability to reason about whether a number is meaningful and to write production code to validate it.
• Track record as an early employee or small, high-output team founder/early hire.

Responsibilities

• Design how to measure AI agents in domains with no ground truth.
• Build verifiers and graders for knowledge work with no ground truth.
• Develop simulated environments and tasks faithful to the real world.
• Determine when tests are sufficient and when a score is meaningful.

Skills

Eval pipelines

Benchmarks

Verifiers

Reward models

RL environments

Synthetic data pipelines

Job description

Litmus is building evaluation tools to measure AI agents performing real work. You will design metrics for domains without ground truth and implement verifiers, graders, and simulation environments that reflect real-world tasks. You will ensure numbers are trustworthy and scalable to customers.

You will ship hands-on evaluation systems, shaping how agents are judged across industries and turning insights into product capabilities. Relocation to San Francisco is supported.