Benchmark Architect for AI Evaluation and Research

Vals AI, Inc.; San Francisco (CA); On-site; USD 150,000 - 260,000; Full time

Benefits offered by this job

Relocation support

Health insurance

Meals provided

Unlimited PTO

Opportunity to publish work

Job summary

Vals AI, Inc. in San Francisco is seeking exceptional researchers and research engineers to design and build the next generation of AI benchmarks that evaluate real-world LLM capabilities.

You will lead development of novel benchmarks shaping how foundation models are built and evaluated. You will collaborate with labs, enterprises, and top industry partners to ensure benchmarks are valid, reliable, and impactful, and you will publish your findings to advance the evaluation research community.

Qualifications

• Advanced research experience demonstrates ability to design rigorous experiments.
• Publication record in reputable venues (NeurIPS, ICML, ACL, EMNLP).
• Strong understanding of experimental design, statistics, and evaluation frameworks.
• Excellent communication of complex ideas to diverse audiences.

Responsibilities

• Design and develop novel, high-impact benchmarks that assess challenging real-world capabilities.
• Conduct research to ensure benchmarks are valid, reliable, and meaningful.
• Collaborate with labs and enterprises to understand evaluation needs.
• Analyze model performance across benchmarks and communicate findings.
• Publish research findings and contribute to the evaluation community.
• Work with infrastructure to implement benchmark designs at scale.
• Stay current with LLM capabilities and evaluation methodologies.

Skills

Advanced research

Publication track record

Experimental design

Python

Communication

Collaboration

Portfolio

Location: SF onsite

Education

Master's degree or PhD in CS/NLP/ML

Undergraduate with strong research background

Tools

Python

Job description

Vals AI, Inc. in San Francisco is seeking exceptional researchers and research engineers to design and build the next generation of AI benchmarks that evaluate real-world LLM capabilities.

You will lead development of novel benchmarks shaping how foundation models are built and evaluated. You will collaborate with labs, enterprises, and top industry partners to ensure benchmarks are valid, reliable, and impactful, and you will publish your findings to advance the evaluation research community.