Head of AI Evaluation & Benchmarks

Vibehackers; San Francisco, Northern (CA, KY); Hybrid; USD 225,000 - 275,000; Full time

Benefits offered by this job

Relocation and transportation support

Health and dental insurance

Lunch and dinner provided

Unlimited PTO

Housing stipend

401(k) plan

Free snacks/coffee/drinks

Equity

Collaboration with leading AI labs

Job summary

Vals AI in San Francisco seeks a Head of Research to define and advance evaluation methodologies for large language models and AI systems. You will set research direction across the portfolio, publish influential work, and recruit a research team while partnering with enterprise customers and lab partners.

The role emphasizes real‑world deployment impact, collaboration with AI labs, and leadership of a high‑caliber team.

Qualifications

• PhD in ML/NLP or equivalent industry track record.
• Deep familiarity with the LLM evaluation landscape, including existing benchmarks and judge‑model approaches.
• Preference for research that influences real‑world deployments rather than gamed benchmarks.
• Strong written and verbal communication for publishing, presenting, and customer engagement.
• Ability to work in‑person in San Francisco.

Responsibilities

• Develop new paradigms for evaluating long-horizon, real‑world tasks that current benchmarks fail to capture.
• Oversee and set direction across Vals’ research portfolio and ongoing projects.
• Publish research and present results to the community, customers, and partners.
• Recruit, mentor, and grow a high-quality research team alongside the founders.
• Collaborate closely with enterprise customers and lab partners to solve practical evaluation problems.

Skills

Research leadership

Experimental design

LLM evaluation

Publication & communication

Team building

Mentorship

Customer-facing

Collaboration

Domain expertise

Learning agility

Ownership

Problem solving

Education

PhD in ML/NLP

Tools

Python

AWS

AWS CDK

Django

React

Job description

Vals AI in San Francisco seeks a Head of Research to define and advance evaluation methodologies for large language models and AI systems. You will set research direction across the portfolio, publish influential work, and recruit a research team while partnering with enterprise customers and lab partners.

The role emphasizes real‑world deployment impact, collaboration with AI labs, and leadership of a high‑caliber team.