Offensive Cyber Eval Architect

Zealot Labs; New York (NY); On-site; USD 140,000 - 200,000; Full time

Job summary

Zealot Labs is seeking an Evals Engineer to own the truth layer for offensive cyber. You will shape the benchmarks, target environments, grading harnesses, instrumentation, metrics, and dashboards that determine what we trust, ship, and build next.

You’ll evaluate AI performance across vulnerability research, exploit development, CTF- and AIxCC-style challenges, and autonomous operations to answer whether non-deterministic agents can perform real offensive security work under realistic

Qualifications

• Experience building LLM, model, or agent evaluation systems, with an understanding of contamination, benchmark overfit, grader drift, prompt sensitivity, brittle scoring, reward hacking, and false confidence.
• Strong systems engineering ability; you can build reliable, reproducible evaluation infrastructure at scale.
• A genuine security background. Hands‑on offensive experience in CTFs, vulnerability research, exploit development, reverse engineering, or AIxCC-style environments is a major advantage.
• The range to read a heap‑corruption writeup and build the harness that determines whether a model actually landed the bug.
• Rigor about measurement. These numbers inform real capability assessments and product and research decisions.

Responsibilities

• Build CTF- and AIxCC-style task suites that reflect real targets and conditions, not toy problems.
• Design eval methodology for non-deterministic agents, including variance across runs, sampling strategies, confidence thresholds, and statistically sound claims of capability.
• Instrument full agent trajectories: tool calls, intermediate state, decision points, failed paths, partial progress, and final outcomes.
• Build graders robust to reward hacking, including trustworthy LLM-as-judge pipelines and scoring for multi-stage exploitation chains.
• Translate results into metrics and dashboards the research team uses to prioritize, and that hold up to scrutiny.

Skills

Evaluation systems

Systems engineering

Offensive security

CTF experience

Job description

Zealot Labs is seeking an Evals Engineer to own the truth layer for offensive cyber. You will shape the benchmarks, target environments, grading harnesses, instrumentation, metrics, and dashboards that determine what we trust, ship, and build next.

You’ll evaluate AI performance across vulnerability research, exploit development, CTF- and AIxCC-style challenges, and autonomous operations to answer whether non-deterministic agents can perform real offensive security work under realistic