Research Engineer, AI Benchmarking for Cybersecurity

Speedrun Talent Network; San Francisco (CA); On-site; USD 180,000 - 260,000; Full time

Benefits offered by this job

Competitive salary

Housing stipend

Relocation support

Health insurance

Meals provided

Unlimited PTO

Publish opportunities

Job summary

Vals AI is building the next generation of AI benchmarks for cybersecurity, designing high-impact evaluations that push what we can measure in foundation models. You will lead the development of novel benchmarks for real-world tasks and risks, shaping how LLMs are evaluated across security domains.

You will collaborate with major labs and enterprise security teams, publish results, and drive benchmarking at scale with the infra team.

Qualifications

• Cybersecurity experience in security engineering, red team, or vulnerability research.
• Systems depth in operating systems, cloud infrastructure and networking.
• Interest in benchmarking; experience building cyber evals or security benchmarks.
• AI-forward: experience using LLMs in security work.
• Strong communication of complex research ideas to technical and non-technical audiences.
• Collaborative experience in research teams and incorporating feedback.
• Location in San Francisco with relocation/transport support.

Responsibilities

• Design and develop novel, high-impact benchmarks for cybersecurity capabilities.
• Conduct research to ensure benchmarks are valid, reliable and meaningful.
• Collaborate with foundation model labs and enterprise security teams to understand needs.
• Analyze model performance across benchmarks and report results.
• Publish research findings and contribute to the research community.
• Work with infrastructure to scale benchmark designs.
• Stay current with developments in LLM cybersecurity and evaluation methods.

Skills

Cybersecurity experience

Systems depth

Benchmarking

AI-ready

Communication

Collaboration

Job description

Vals AI is building the next generation of AI benchmarks for cybersecurity, designing high-impact evaluations that push what we can measure in foundation models. You will lead the development of novel benchmarks for real-world tasks and risks, shaping how LLMs are evaluated across security domains.

You will collaborate with major labs and enterprise security teams, publish results, and drive benchmarking at scale with the infra team.