Staff Engineer, Evaluation Systems

Liquid AI; San Francisco (CA); Hybrid; USD 150,000 - 190,000; Full time

Benefits offered by this job

Equity

Unlimited PTO

Refill Days

Comprehensive health coverage

Paid leave

Job summary

Liquid AI in San Francisco is seeking a Member of Technical Staff - Evaluations to own the internal eval suite and its impact on post-training decisions. You will work closely with the Post-Training team to design, implement, and run reliable benchmarks for real-world model behavior.

You will release open-source evals, publish findings, and help scale evaluation across teams, ensuring reproducibility, contamination checks, and transparent reporting in a hybrid work environment.

Qualifications

• Strong ability to translate capability descriptions into robust, general-purpose evaluations.
• Past benchmarks or evals built from scratch (benchmark/paper/LM eval harness contributions).
• Fluency with contamination, saturation, judge bias, and variance concepts.
• Hands-on experience running evals in a team (engineering/DevOps).
• Skill at debugging scores by inspecting model outputs before trusting numbers.
• Software engineering chops to own end-to-end eval infrastructure.

Responsibilities

• Build and maintain the internal eval suite and infrastructure at scale.
• Design held-out evals for agentic workflows, computer use, and cybersecurity with documented checks.
• Create evals for small edge models where frontier-model evals don't apply.
• Read generated outputs to diagnose unexpected scores before actions.
• Translate customer and open-source feedback into data/training/evaluation environments; collaborate with teams.
• Release open-source evals, publish papers, and write failure-mode reviews.

Skills

Eval design

Benchmarking

Reproducibility

Contamination handling

DevOps experience

Open-source evals

Tools

lm-eval-harness

lighteval

OLMES

HELM

Inspect

Job description

Liquid AI in San Francisco is seeking a Member of Technical Staff - Evaluations to own the internal eval suite and its impact on post-training decisions. You will work closely with the Post-Training team to design, implement, and run reliable benchmarks for real-world model behavior.

You will release open-source evals, publish findings, and help scale evaluation across teams, ensuring reproducibility, contamination checks, and transparent reporting in a hybrid work environment.