Founding Engineer, Applied AI Evaluation

US Health Partners, LLC; San Francisco (CA); On-site; USD 120,000 - 180,000; Full time

Benefits offered by this job

Founding equity

Competitive cash

Office in FiDi

Meals provided

Healthcare coverage

Equinox membership or fitness stipend

AI tools budget

Relocation support

Job summary

Litmus is building evaluation tools to measure AI agents performing real work. You will design metrics for domains without ground truth and implement verifiers, graders, and simulation environments that reflect real-world tasks. You will ensure numbers are trustworthy and scalable to customers.

You will ship hands-on evaluation systems, shaping how agents are judged across industries and turning insights into product capabilities. Relocation to San Francisco is supported.

Qualifications

• Experience building evals, benchmarks, verifiers, or RL environments for LLMs or agents.
• Ability to reason about whether a number is meaningful and to write production code to validate it.
• Track record as an early employee or small, high-output team founder/early hire.

Responsibilities

• Design how to measure AI agents in domains with no ground truth.
• Build verifiers and graders for knowledge work with no ground truth.
• Develop simulated environments and tasks faithful to the real world.
• Determine when tests are sufficient and when a score is meaningful.

Skills

Eval pipelines

Benchmarks

Verifiers

Reward models

RL environments

Synthetic data pipelines

Job description

About Litmus

AI agents are starting to do real work across the economy, and the companies deploying them have no reliable way to know whether they can trust them. Litmus is building the system that lets every enterprise measure how its agents perform, understand why they fail, and make them better, starting with evaluation and building toward agents that improve on their own.

Litmus was founded in August 2026 by Sidharth Srinivasan, who built and led AI evaluation at Box after Box's CEO recruited him out of Stanford CS. We've raised $5M from Pear, BoxGroup, Workbench, and Emergence, alongside the CEO of Box, the co-founder of Dropbox, and senior executives from OpenAI, Anthropic, and Google.

The role

Figure out what "good" means for AI agents doing real work, and prove it.

You'll design how we measure agents in domains where there's no answer key, and make sure the numbers can be trusted. It's research-leaning, hands-on work that ships to customers.

You'll work on:

• Verifiers and graders for knowledge work with no ground truth
• Making simulated environments and tasks faithful to the real world
• Knowing when you've tested enough, and when a score means something
• Turning what works for one customer into something the product does automatically

Why it's exciting:

• Nobody has written the playbook for measuring agents in real work yet. You'll write it.
• Real agents in high-stakes domains, not academic benchmarks.
• Measurement, not capability: whether agents are actually useful, and how anyone would know.

You might be a fit if

• You've built evals, benchmarks, verifiers, reward models, RL environments, or synthetic data pipelines for LLMs or agents
• You think like a researcher about whether a number means anything, and you write production code to find out
• You've chosen risk before: founder, founding engineer, early employee, or a small, high-output team
• You typically have 2–6 years of experience

Nice to have

• Experience in non-code domains like legal, finance, healthcare, or enterprise work
• Post-training experience (SFT, RLHF, RLVR)
• Statistics for evaluation, benchmark design, or published eval research

Probably not a fit

• Eval dashboards or observability tooling with no hands-on eval design
• Model training with little interest in evaluation

What we offer

• Founding equity and competitive cash
• One of the first hires, working directly with the founder and customers from day one
• A say in how agent quality gets measured across the industry
• A beautiful office in FiDi
• Breakfast, lunch, and dinner
• Medical, dental, and vision coverage
• Equinox membership or a fitness stipend
• Essentially unlimited budget for AI tools and tokens
• Relocation support if you're moving to San Francisco