Member of Technical Staff - Research, Cyber

Vals AI, Inc.; San Francisco (CA); On-site; USD 150,000 - 230,000; Full time

Benefits offered by this job

Relocation support

Health/dental insurance

Lunch and dinner provided

Unlimited PTO

$1,500 housing stipend (within one‑m​i

Housing stipend within 1 mile radius

Collaborate with leading AI labs

Opportunity to publish

Job summary

Vals AI, Inc. seeks an exceptional researcher to design and build the next generation of AI benchmarks for cybersecurity. You will create high-impact evaluations that push what we can measure in foundation models, shaping how the world evaluates AI systems.

The role offers relocation and in-person collaboration in San Francisco with leading labs and major institutions. You will lead design and development of novel benchmarks, collaborate with labs and security teams, and publish findings to

Qualifications

• Hands-on security experience (e.g., security engineer, pentester, red teamer, or vulnerability researcher).
• Strong grounding in operating systems, cloud infrastructure and networking.
• Interest in benchmarking and prior cyber evals, CTFs or security benchmarks preferred.
• AI-forward: use LLMs in security work.
• Clear communication of complex research ideas to technical and non-technical audiences.
• Experience collaborating in research teams and integrating feedback.
• Location: in-person SF team with relocation/transport support.

Responsibilities

• Design and develop novel, high-impact benchmarks for cybersecurity real-world capabilities.
• Conduct research to ensure benchmarks are valid, reliable, and meaningful.
• Collaborate with foundation model labs and enterprise security teams to understand evolving evaluation needs.
• Analyze model performance across benchmarks and articulate results.
• Publish research findings and contribute to security and evaluation communities.
• Work with the infrastructure team to implement benchmark designs at scale.
• Stay updated with latest developments in LLM cyber capabilities and evaluation methodologies.

Skills

Cybersecurity

Systems depth

Benchmarking

AI-forward

Communication

Collaboration

San Francisco location

Tools

Python

Django

AWS

CDK

React

Job description

About the role

We are looking for an exceptional researcher to design and build the next generation of AI benchmarks for cybersecurity. You will create high-impact, challenging evaluations that push the boundaries of what we can measure in foundation models. This role is ideal for someone with deep research expertise who wants to see their work directly influence how the world evaluates AI systems.

You will lead the design and development of novel benchmarks that assess real-world capabilities and risks of LLMs. Our benchmarks shape how foundation models are developed and generative AI applications are built. We work with every major foundation model lab - along with leading financial institutions and the application-layer companies pushing the frontier forward. Our work has been featured by the Wall Street Journal, New York Times, Washington Post, and Bloomberg.

We are building the standard for evaluating the ability of LLMs to perform real-world tasks. You will be at the forefront of defining what that standard looks like.

What you'll do

• Design and develop novel, high-impact benchmarks that assess challenging real-world capabilities for cybersecurity

• Conduct research to ensure our benchmarks are valid, reliable, and meaningful

• Collaborate with foundation model labs and enterprise security teams to understand evolving evaluation needs

• Analyze model performance across benchmarks and articulate results

• Publish research findings and contribute to the broader security and evaluation research communities

• Work closely with the infrastructure team to implement your benchmark designs at scale

• Stay current with the latest developments in LLM cyber capabilities and evaluation methodologies

Requirements

• Cybersecurity experience: Hands-on work in security, e.g., as a security engineer, penetration tester, red teamer or vulnerability researcher. You may have found and reported vulnerabilities through bug bounty programs or CVEs.

• Systems depth: Strong grounding in operating systems, cloud infrastructure and networking.

• Interest in benchmarking: You care about measuring capability rigorously, not just demonstrating it. Ideally, you've built cyber evals, CTF challenges or security benchmarks before.

• AI-forward: You already use LLMs in your security work and want to work with them every day.

• Communication: Ability to clearly communicate complex research ideas to both technical and non-technical audiences.

• Collaboration: Experience working in research teams and integrating feedback.

• Location: We are an in-person team in San Francisco and will support relocation or transportation as needed.

Nice to haves

• You've previously worked as a security engineer at a security company or on an in-house security team, and/or you have experience on an AI or ML team at a security company

• Strong ties to the security community: CTF teams, conference talks, open-source security tools, or research collaborations, published security research, papers or well-read technical write-ups

What we offer

• Highly competitive salary and meaningful ownership. Excellence is well rewarded.

• Relocation and transportation support

• Health/dental insurance coverage

• Lunch and dinner provided, free snacks/coffee/drinks

• Unlimited PTO

• $1,500 housing stipend (within one-mile radius)

• Collaborate with leading AI labs and work on the front lines of AI development and evaluations

• Opportunity to publish and present your work

About us

Founding team: The core methodology behind this platform comes from NLP evaluation research we conducted at Stanford. We raised a $5M seed from some of the top institutional and angel investors in the valley. Our team has prior work experience at NVIDIA, Meta, Microsoft, Palantir and HRT. Collectively, we have over 300 citations in our published work. Our early team includes Stanford PhDs, ex-Jane Street quants, and the first designer at Snorkel.

Tech stack: We use Python for most things at Vals AI. Our platform is built on Django, with a React frontend. All of the infra is on AWS using CDK for IaC.

What We're Looking For

• Learning velocity: The role encompasses a wide variety of tasks. Rather than expecting you to be an expert on Day 1, we are looking for someone who can learn new skills and technologies quickly.

• Ownership: Working in a small, talent-dense team, we expect everyone to show initiative to build where it's needed, not where it's asked. We strive for autonomy over consensus.

• Intensity: The LLM landscape is constantly changing. Foundation model labs are continuously pushing the frontier. The unicorn companies that will emerge from this technology shift are being built now. Those that win will have an incredibly high speed of execution.

• Solution-oriented mindset: We're looking for people who see opportunities to craft solutions at each juncture, not those who pass hard problems to others or admit defeat.

Vals in the Media

• AI tools mostly fumble basic financial tasks, study finds

• The Winners (and Losers) of This New Vibe-Coding Benchmark Will Surprise You

• OpenAI’s Less-Flashy Rival Might Have a Better Business Model

• Meta Announces New AI Model in Major Test of Company’s Ambitions

• DeepSeek’s Sequel Set to Extend China’s Reach in Open-Source A.I.