Staff AI Benchmark Architect

Vals AI, Inc.; San Francisco (CA); Sur place; USD 150 000 - 230 000; Plein temps

Avantages offerts par ce poste

Relocation support

Health/dental insurance

Lunch and dinner provided

Unlimited PTO

Housing stipend (within SF)

Résumé du poste

Vals AI, Inc. is seeking exceptional researchers and research engineers to design and build AI benchmarks that push the boundaries of evaluation for foundation models.

You will lead the development of novel benchmarks and collaborate with labs and enterprises to understand evolving evaluation needs. You will publish findings and contribute to the broader evaluation research community while working with an in-house infrastructure team to scale benchmark deployments.

Qualifications

• Advanced research experience at graduate or post-graduate level.
• Publication track record in reputable venues (NeurIPS/ICML/ACL/EMNLP).
• Strong understanding of experimental design and evaluation frameworks.
• Proficiency in Python for research and experimentation.
• Clear communication of complex ideas to technical and non-technical audiences.
• Experience working in research teams and integrating feedback.

Responsabilités

• Design and develop novel benchmarks that assess real-world LLM capabilities.
• Conduct research to ensure benchmarks are valid, reliable, and meaningful.
• Collaborate with labs and enterprises to understand evolving evaluation needs.
• Analyze model performance across benchmarks and articulate results.
• Publish research findings and contribute to the evaluation community.
• Work with the infrastructure team to implement designs at scale.
• Stay current with developments in LLM capabilities and evaluation methodologies.

Connaissances

Advanced research

Publication track record

Research methodology

Python for research

Communication

Collaboration

Location in SF

Portfolio

Formation

Master's degree or PhD in CS/NLP/ML

Undergraduate with strong research background

Outils

Python

Description du poste

Vals AI, Inc. is seeking exceptional researchers and research engineers to design and build AI benchmarks that push the boundaries of evaluation for foundation models.

You will lead the development of novel benchmarks and collaborate with labs and enterprises to understand evolving evaluation needs. You will publish findings and contribute to the broader evaluation research community while working with an in-house infrastructure team to scale benchmark deployments.