Lead AI Cybersecurity Benchmark Architect

Vals; San Francisco (CA); On-site; USD 170,000 - 230,000; Full time

Job summary

Vals AI is seeking an exceptional researcher to design and build next-generation AI benchmarks for cybersecurity in San Francisco. You will lead the creation of challenging evaluations that measure real-world capabilities and risks of LLMs, guiding how foundation models are developed and evaluated.

You will design novel benchmarks, validate their rigor, and collaborate with labs and security teams to understand evolving needs.

Qualifications

• Hands-on cybersecurity experience (security engineer, pentester, red teamer, or vulnerability researcher).
• Strong grounding in OS, cloud infra, and networking.
• Interest in benchmarking and building cyber evals or benchmarks.
• AI-forward: use LLMs in security work and apply them daily.
• Clear communication of complex research to technical and non-technical audiences.
• Experience in research teams and integrating feedback.

Responsibilities

• Design and develop high-impact benchmarks for cybersecurity real-world capabilities.
• Conduct research to ensure benchmarks are valid, reliable, and meaningful.
• Collaborate with foundation model labs and enterprise security teams.
• Analyze model performance across benchmarks and articulate results.
• Publish research findings and contribute to the evaluation community.
• Collaborate with infrastructure team to scale benchmark implementations.
• Stay updated with latest LLM cyber capabilities and evaluation methods.

Skills

Cybersecurity experience

Systems depth

AI–forward

Communication

Collaboration

Tools

Python

Django

React

AWS CDK

Job description

Vals AI is seeking an exceptional researcher to design and build next-generation AI benchmarks for cybersecurity in San Francisco. You will lead the creation of challenging evaluations that measure real-world capabilities and risks of LLMs, guiding how foundation models are developed and evaluated.

You will design novel benchmarks, validate their rigor, and collaborate with labs and security teams to understand evolving needs.