Member of Technical Staff: Agents, Evaluations, and Environments

Asymmetric Security

New York (NY)

Hybrid

USD 180,000 - 240,000

Full time

5 days ago
Be an early applicant

Get a reply from this employer

Your resume and cover letter, tailored to exactly what they’re hiring for.

Get past ATS filters

Job summary

Asymmetric is seeking a Member of Technical Staff to advance agents, evaluations, and environments in London/New York. You will design longer, harder investigations, build scalable evaluation suites, and work with responders to ground truth tasks and scoring.

This role emphasizes rigorous, privacy-preserving environments and measurable model improvement. You will contribute to architecture, tooling, and experiments to push state-of-the-art AI cyberdefense capabilities and help scale our

Qualifications

  • Experience shipping LLM agents or LLM-powered systems beyond prototypes.
  • Ability to design robust evaluations and understand leakage, reward hacking, and overfitting.
  • Fluent in Python and able to build the infrastructure your experiments need.
  • Strong ability to reason about experiment design and statistical significance.
  • Ability to interpret model trajectories to form hypotheses from what you see.
  • Deep motivation by our mission.

Responsibilities

  • Own the design and iteration of our investigation agents, from prompts and tools to architecture and orchestration.
  • Build and maintain the evaluation platform: suite governance, regression tracking, trajectory review, and fair comparisons across models and agent versions.
  • Build environment pipelines that convert real cases into graded tasks, and ensure graders reward the right behaviour and can’t be gamed.
  • Run experiments and write them up clearly: what changed, what it bought, and how sure we are.
  • Work closely with incident responders to capture expert judgement as tasks, rubrics and ground truth.
  • Shape the research direction alongside the founding team.

Skills

LLM agents
Evaluation design
Python
Statistics
Model analysis
Research direction

Job description

Member of Technical Staff: Agents, Evaluations, and Environments

London / New York · Full Time

About Asymmetric

Asymmetric is the first AI-native Digital Forensics and Incident Response company. Our platform has helped incident responders handle hundreds of cyber attacks. Building on that experience, we create realistic investigation environments and evaluations to train and measure AI on real defensive work, both for our own agents and for our partners’ models.

Our mission is to accelerate AI cyberdefense to address the security challenges of the AGI era.

We believe that automated digital forensics, i.e. deep security investigations, will become ubiquitous in a world with abundant intelligence. Read more about our vision.

About the role

Our investigation agent already works real incidents alongside our responders. It queries evidence, runs analysis in a sandbox, and reports findings an analyst can trace back to the source. Every case it works teaches us more about what deep forensic reasoning needs from a model.

We’re hiring for the research-leaning side of that work: designing better agents, and building the environments and evaluations that tell us, rigorously, whether they’re getting better. We turn real investigations into realistic, privacy-preserving environments with verifiable outcomes, and use them to measure and optimise models for our own product and with our partners. For example, we need to:

  • Design agent architectures for longer, harder investigations: tools, skills, planning, sub-agents, and how an agent works through millions of log rows without losing the thread.
  • Build evaluation suites from real cases, with deterministic graders, hidden ground truth, and statistics that separate a real improvement from noise (pass@k, confidence intervals, paired comparisons).
  • Turn investigations into reproducible environments: obfuscate real evidence so it stays realistic but carries no customer data, package it with tasks and reward rules, and run it at scale in isolated sandboxes.
  • Study model behaviour on defensive security work: where frontier and open models fail, why, and what data and environments close the gap.
Illustrative responsibilities
  • Own the design and iteration of our investigation agents, from prompts and tools to architecture and orchestration.
  • Build and maintain the evaluation platform: suite governance, regression tracking, trajectory review, and fair comparisons across models and agent versions.
  • Build environment pipelines that convert real cases into graded tasks, and make sure the graders reward the right behaviour and can’t be gamed.
  • Run experiments and write them up clearly: what changed, what it bought, and how sure we are.
  • Work closely with our incident responders to capture expert judgement as tasks, rubrics and ground truth.
  • Shape the research direction alongside the founding team.
You may be a fit if you:
  • Have built and shipped LLM agents or LLM-powered systems beyond a prototype.
  • Can design evaluations that measure what matters, and know how evals mislead: leakage, reward hacking, noisy baselines, and overfitting to a suite.
  • Are fluent in Python and comfortable building the infrastructure your experiments need.
  • Think quantitatively about experiment design and statistical significance.
  • Read model trajectories closely and form hypotheses from what you see.
  • Are deeply motivated by our mission.

You’re encouraged to apply even if you don’t believe you meet every listed qualification.

Nice to Have
  • Experience building RL environments, reward functions or verifiers, or post-training language models.
  • Research experience in ML, NLP or agents: publications, open-source work or strong independent projects.
  • Experience with sandboxed execution, containerised eval harnesses, or running evals at scale on cloud infrastructure.
  • Data anonymisation, synthetic data generation, or privacy-preserving data pipelines.
  • A background in cybersecurity, incident response or digital forensics.
Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Staff AI Agent Architect: Evaluations & Environments
Staff AI Agent Architect: Evaluations & Environments

Asymmetric Security • New York (NY)

Hybrid
USD 140,000 - 190,000
Member of Technical Staff: Software Engineering
Member of Technical Staff: Software Engineering

Asymmetric Security • New York (NY)

Hybrid
USD 180,000 - 230,000
Staff Software Engineer, AI-Driven Forensics Platform
Staff Software Engineer, AI-Driven Forensics Platform

Asymmetric Security • New York (NY)

Hybrid
USD 140,000 - 210,000
MTS - Research (Cybersecurity)
MTS - Research (Cybersecurity)

Collinear AI, Inc. • Sunnyvale (CA)

On-site
USD 140,000 - 190,000
Staff Engineer: AI Agents, Evaluations & Environments
Staff Engineer: AI Agents, Evaluations & Environments

Asymmetric Security • New York (NY)

Hybrid
USD 180,000 - 240,000
Evals Lead
Evals Lead

Aslan • Washington

On-site
USD 120,000 - 180,000
Artificial Intelligence Researcher (Cybersecurity) - PhD
Artificial Intelligence Researcher (Cybersecurity) - PhD

Stealth AI Startup • San Francisco (CA)

On-site
USD 120,000 - 180,000
AI Engineer
AI Engineer

Aslan • Washington

On-site
USD 150,000 - 210,000
Staff / Principal Defensive Security Engineer, Frontier Cyber Models
Staff / Principal Defensive Security Engineer, Frontier Cyber Models

Tykhe Inc • San Francisco (CA)

On-site
USD 180,000 - 270,000
Member of Technical Staff (Applied AI Engineer, Agent Capabilities)
Member of Technical Staff (Applied AI Engineer, Agent Capabilities)

Perplexity • San Francisco (CA)

On-site
USD 190,000 - 230,000