LLM Benchmark Engineer — Elevate Model Evaluation

Vals AI, Inc.; San Francisco (CA); Presencial; USD 180 000 - 240 000; Tempo integral

Vantagens oferecidas por esta oferta de emprego

Relocation support

Health insurance

Lunch provided

401K

Unlimited PTO

Housing stipend

Resumo da oferta

Vals AI, Inc. is seeking engineers to own and expand our leaderboards, evaluating new LLM models against benchmarks across law, tax, coding, finance, and more. You will analyze error modes, compare strengths and weaknesses, and collaborate with a communications team to share results.

The role emphasizes strong Python engineering, sprint-based work, and close collaboration with both open-source and foundation model labs. Relocation and in-person SF work are supported as needed.

Qualificações

• Familiarity with the space of large language models and current leading models.
• Strong engineering fundamentals and ability to ship quickly with high quality.
• Significant Python experience in a professional setting.
• Experience working in sprints, Git workflows, and code reviews.
• Willingness to work long hours during model releases and meet tight deadlines.

Responsabilidades

• Evaluate new LLM model releases across the Vals AI benchmarks.
• Collaborate with open-source and closed-source labs to assess model performance.
• Use Docent to analyze failure modes and patterns in model output.
• Publish findings with our social media team.
• Add new models and maintain integrations in the model library.
• Maintain and improve benchmark infrastructure (agentic and non-agentic).

Conhecimentos

Python

LLM knowledge

Engineering fundamentals

Team collaboration

Git & CI

Ferramentas

Docent

Django

AWS

CDK

Descrição da oferta de emprego

Vals AI, Inc. is seeking engineers to own and expand our leaderboards, evaluating new LLM models against benchmarks across law, tax, coding, finance, and more. You will analyze error modes, compare strengths and weaknesses, and collaborate with a communications team to share results.

The role emphasizes strong Python engineering, sprint-based work, and close collaboration with both open-source and foundation model labs. Relocation and in-person SF work are supported as needed.