Staff AI Agent Architect: Evaluations & Environments

Asymmetric Security; New York (NY); Hybrid; USD 140,000 - 190,000; Full time

Job summary

Asymmetric Security is seeking a Member of Technical Staff focused on Agents, Evaluations, and Environments. You will design advanced LLM agents, build evaluation suites with deterministic graders, and develop scalable, privacy-preserving environments to measure and improve defensive AI models.

You will work closely with incident responders to capture expert judgement, shape research directions, and push the boundaries of forensic reasoning in AI-enabled security.

Qualifications

• Fluent in Python and able to build experiment infrastructure.
• Ability to design evaluations that measure meaningful progress and guard against leakage.
• Experience with sandboxed evaluation environments and scalable test harnesses.

Responsibilities

• Own design and iteration of investigation agents, including prompts, tools, and architecture.
• Build and maintain the evaluation platform with governance, regression tracking and fair model comparisons.
• Create environment pipelines that convert real cases into graded tasks with robust ground truth.

Skills

Python

LLM agents

Evaluation design

Tools

Sandbox

Eval harness

Cloud infra

Job description

Member of Technical Staff: Agents, Evaluations, and Environments

Asymmetric is the first AI-native Digital Forensics and Incident Response company. Our platform has helped incident responders handle hundreds of cyber attacks. Building on that experience, we create realistic investigation environments and evaluations to train and measure AI on real defensive work, both for our own agents and for our partners’ models.

Our mission is to accelerate AI cyberdefense to address the security challenges of the AGI era.

We believe that automated digital forensics, i.e. deep security investigations, will become ubiquitous in a world with abundant intelligence. Read more about our vision .

About the role

Our investigation agent already works real incidents alongside our responders. It queries evidence, runs analysis in a sandbox, and reports findings an analyst can trace back to the source. Every case it works teaches us more about what deep forensic reasoning needs from a model.

We’re hiring for the research-leaning side of that work: designing better agents, and building the environments and evaluations that tell us, rigorously, whether they’re getting better. We turn real investigations into realistic, privacy-preserving environments with verifiable outcomes, and use them to measure and optimise models for our own product and with our partners. For example, we need to:

• Design agent architectures for longer, harder investigations: tools, skills, planning, sub-agents, and how an agent works through millions of log rows without losing the thread.
• Build evaluation suites from real cases, with deterministic graders, hidden ground truth, and statistics that separate a real improvement from noise (pass@k, confidence intervals, paired comparisons).
• Turn investigations into reproducible environments: obfuscate real evidence so it stays realistic but carries no customer data, package it with tasks and reward rules, and run it at scale in isolated sandboxes.
• Study model behaviour on defensive security work: where frontier and open models fail, why, and what data and environments close the gap.

Illustrative responsibilities

• Own the design and iteration of our investigation agents, from prompts and tools to architecture and orchestration.
• Build and maintain the evaluation platform: suite governance, regression tracking, trajectory review, and fair comparisons across models and agent versions.
• Build environment pipelines that convert real cases into graded tasks, and make sure the graders reward the right behaviour and can’t be gamed.
• Run experiments and write them up clearly: what changed, what it bought, and how sure we are.
• Work closely with our incident responders to capture expert judgement as tasks, rubrics and ground truth.
• Shape the research direction alongside the founding team.

You may be a fit if you:

• Have built and shipped LLM agents or LLM-powered systems beyond a prototype.
• Can design evaluations that measure what matters, and know how evals mislead: leakage, reward hacking, noisy baselines, and overfitting to a suite.
• Are fluent in Python and comfortable building the infrastructure your experiments need.
• Think quantitatively about experiment design and statistical significance.
• Read model trajectories closely and form hypotheses from what you see.
• Are deeply motivated by our mission.

You’re encouraged to apply even if you don’t believe you meet every listed qualification.

Nice to Have

• Experience building RL environments, reward functions or verifiers, or post-training language models.
• Research experience in ML, NLP or agents: publications, open-source work or strong independent projects.
• Experience with sandboxed execution, containerised eval harnesses, or running evals at scale on cloud infrastructure.
• Data anonymisation, synthetic data generation, or privacy-preserving data pipelines.
• A background in cybersecurity, incident response or digital forensics.

Apply for Member of Technical Staff: Agents, Evaluations, and Environments

First name*

Last name*

Email*

LinkedIn profile URL*

CV* Click to upload file or drag-and-drop. Max 25 MB