Member of Technical Staff - Research

Vals AI, Inc.; San Francisco (CA); On-site; USD 150,000 - 260,000; Full time

Benefits offered by this job

Relocation support

Health insurance

Meals provided

Unlimited PTO

Opportunity to publish work

Job summary

Vals AI, Inc. in San Francisco is seeking exceptional researchers and research engineers to design and build the next generation of AI benchmarks that evaluate real-world LLM capabilities.

You will lead development of novel benchmarks shaping how foundation models are built and evaluated. You will collaborate with labs, enterprises, and top industry partners to ensure benchmarks are valid, reliable, and impactful, and you will publish your findings to advance the evaluation research community.

Qualifications

• Advanced research experience demonstrates ability to design rigorous experiments.
• Publication record in reputable venues (NeurIPS, ICML, ACL, EMNLP).
• Strong understanding of experimental design, statistics, and evaluation frameworks.
• Excellent communication of complex ideas to diverse audiences.

Responsibilities

• Design and develop novel, high-impact benchmarks that assess challenging real-world capabilities.
• Conduct research to ensure benchmarks are valid, reliable, and meaningful.
• Collaborate with labs and enterprises to understand evaluation needs.
• Analyze model performance across benchmarks and communicate findings.
• Publish research findings and contribute to the evaluation community.
• Work with infrastructure to implement benchmark designs at scale.
• Stay current with LLM capabilities and evaluation methodologies.

Skills

Advanced research

Publication track record

Experimental design

Python

Communication

Collaboration

Portfolio

Location: SF onsite

Education

Master's degree or PhD in CS/NLP/ML

Undergraduate with strong research background

Tools

Python

Job description

About the Role

We are looking for exceptional researchers and research engineers to design and build the next generation of AI benchmarks. You will create high-impact, challenging evaluations that push the boundaries of what we can measure in foundation models. This role is perfect for someone with deep research expertise who wants to see their work directly influence how the world evaluates AI systems.

You will lead the design and development of novel benchmarks that assess real-world capabilities of LLMs. Our benchmark shapes how foundation models are developed and generative AI applications are built. We work with every major foundation model lab – along with leading financial institutions and the application-layer companies pushing the frontier forward. Our work has been featured by the Wall Street Journal, Washington Post, and Bloomberg.

We are building the standard for evaluating the ability of LLMs to perform real-world tasks. You will be at the forefront of defining what that standard looks like.

What You'll Do

• Design and develop novel, high-impact benchmarks that assess challenging real-world capabilities
• Conduct research to ensure our benchmarks are valid, reliable, and meaningful
• Collaborate with foundation model labs and enterprises to understand evaluation needs
• Analyze model performance across benchmarks and communicate findings
• Publish research findings and contribute to the broader evaluation research community
• Work closely with the infrastructure team to implement your benchmark designs at scale
• Stay current with the latest developments in LLM capabilities and evaluation methodologies

Requirements

• Advanced research experience: Master's degree or PhD in Computer Science, NLP, Machine Learning, or related field. Undergrads with very strong research backgrounds may also be considered.
• Publication track record: Published papers in reputable venues (NeurIPS, ICML, ACL, EMNLP, etc.) with focus on NLP, ML evaluation, or benchmarking
• Research methodology: Strong understanding of experimental design, statistical analysis, and evaluation frameworks
• Technical skills: Proficiency in Python for research and experimentationCommunication: Ability to clearly communicate complex research ideas to both technical and non-technical audiences
• Collaboration: Experience working in research teams and integrating feedback
• Portfolio: Demonstrated track record of impactful research work
• Location: We are an in-person team based in San Francisco. We will support your relocation or transportation as needed.

Nice to Haves

• Experience specifically in LLM evaluation or benchmarking research
• Familiarity with foundation model architectures and capabilities
• Experience working with industry partners or in applied research settings
• Background in areas like human-computer interaction, psychology, or domain-specific evaluation
• Experience at early-stage startups or research labs
• Contributions to open-source evaluation tools or datasets

What We Offer

• Highly competitive salary and meaningful ownership. Excellence is well rewarded.
• Relocation and transportation support
• Health/dental insurance coverage
• Lunch and dinner provided, free snacks/coffee/drinks
• Unlimited PTO
• Opportunity to publish and present your work