LLM Evaluations Engineer — Benchmark Leaderboards

Vals AI; San Francisco (CA); On-site; USD 180,000 - 240,000; Full time

Benefits offered by this job

Health/dental insurance coverage

Relocation support

Lunch and dinner provided

401K plan

Unlimited PTO

Job summary

Vals AI in San Francisco is seeking engineers to own leaderboards that evaluate LLMs across tasks including law, tax, coding, finance, and more. You will test and benchmark new models as released, analyze error modes, and work with our communications team to publish results.

You will collaborate with open-source and closed-source labs, maintain the model library and integrations, and help improve the benchmarking infrastructure as new models launch in fast sprints.

Qualifications

• Proficient in Python with professional experience.
• Strong familiarity with large language models and current benchmarks.
• Solid engineering fundamentals and ability to ship high-quality code.

Responsibilities

• Evaluate new LLM releases across Vals AI benchmarks.
• Assess model performance with open-source and closed-source labs.
• Use Docent to analyze failure modes and patterns.
• Collaborate with social media for results publication.
• Add models and maintain library integrations.
• Improve benchmark infrastructure for agentic and non-agentic runs.
• Collaborate with the research team on new benchmarks.
• Adapt to sprint-driven release cycles.

Skills

Python

LLM familiarity

Engineering fundamentals

Team collaboration

Benchmarking

Tools

Django

AWS

Git

React

Job description

Vals AI in San Francisco is seeking engineers to own leaderboards that evaluate LLMs across tasks including law, tax, coding, finance, and more. You will test and benchmark new models as released, analyze error modes, and work with our communications team to publish results.

You will collaborate with open-source and closed-source labs, maintain the model library and integrations, and help improve the benchmarking infrastructure as new models launch in fast sprints.