LLM Benchmark Engineer — Lead Leaderboard Insights

Vals AI, Inc.; San Francisco (CA); On-site; USD 150,000 - 190,000; Full time

Benefits offered by this job

Relocation and transportation support

Health/dental insurance coverage

Lunch and dinner provided

401K plan

Unlimited PTO

Job summary

Vals AI, Inc. in San Francisco is seeking strong engineers to own the leaderboards that evaluate real-world tasks by LLMs.

You will test and benchmark new models across domains like law, tax, coding, finance and more, analyzing error modes and collaborating with the communications team to publish results. The role involves working with major labs and institutions, building model library integrations, and maintaining benchmarks infrastructure in a fast, sprint-driven environment.

Qualifications

• Familiarity with the LLMs and leading models in the space.
• Strong engineering fundamentals; able to build and ship quickly with high quality.
• Significant experience in Python in a professional setting.
• Team collaboration in sprints, Git workflows, and pull request reviews.
• In-person in San Francisco; relocation support may be provided.

Responsibilities

• Evaluate new LLM model releases across the Vals AI benchmarks.
• Work with open-source and closed-source labs to evaluate model performance.
• Use Docent to analyze common failure modes and patterns in model performance.
• Collaborate with social media to post findings and results.
• Add new models and maintain integrations in the model library.
• Improve and maintain benchmarking infrastructure (agentic and non-agentic).
• Partner with research to create new benchmarks.

Skills

Python

LLMs familiarity

Engineering fundamentals

Team collaboration

Tools

Docent

Git workflows

Django

React

AWS CDK

Job description

Vals AI, Inc. in San Francisco is seeking strong engineers to own the leaderboards that evaluate real-world tasks by LLMs.

You will test and benchmark new models across domains like law, tax, coding, finance and more, analyzing error modes and collaborating with the communications team to publish results. The role involves working with major labs and institutions, building model library integrations, and maintaining benchmarks infrastructure in a fast, sprint-driven environment.