Staff Product Manager: Agentic AI Evaluations

Twenty; Arlington (VA); On-site; USD 140,000 - 170,000; Full time

Benefits offered by this job

Health, dental, and vision plans

401(k) with company match

Paid time off and holidays

Life/AD&D, disability coverage

Job summary

Twenty builds AI-enabled technology for mission-critical cyber operations. We're seeking a Staff Product Manager to help make agentic AI systems measurable, reliable, and continuously improvable.

You’ll partner with AI and engineering teams to shape evaluation capabilities—from defining agent performance and product quality to turning results into useful workflows and product decisions. The strongest candidates will bring experience building or managing agentic systems, either by directly

Qualifications

• Must have strong product experience with AI or agentic systems and experience building or managing agentic systems through product work.
• Experience directly building evaluations for agents or the evaluation product, tooling, or system used to assess them.
• Ability to translate technical evaluation needs into product requirements, priorities, and workflows in partnership with engineering and AI teams.
• Ability to define or operationalize meaningful measures of agent performance and product quality.
• Strong communication, collaboration, and product judgment in an evolving, technically complex environment.

Responsibilities

• Own or support the product roadmap for agentic AI evaluations, translating product-area goals into clear priorities and outcomes.
• Partner closely with engineering and AI teams to understand evaluation needs, technical constraints, and user workflows.
• Define how agent performance and product quality should be assessed, including useful measures of task success, reliability, and quality.
• Turn technical evaluation needs into clear product requirements, workflows, and acceptance criteria.
• Help operationalize evaluation workflows and success metrics so teams can consistently assess agent behavior and product changes.
• Use evaluation results and feedback to identify gaps, inform product decisions, and improve the agent experience over time.

Skills

Strong product experience with AI or

Experience building evaluations for AI

Translate technical evaluation needs

Define meaningful measures of agent

Strong communication and collaboration

Cybersecurity experience (nice to have

Job description

Twenty builds AI-enabled technology for mission-critical cyber operations. We're seeking a Staff Product Manager to help make agentic AI systems measurable, reliable, and continuously improvable.

You’ll partner with AI and engineering teams to shape evaluation capabilities—from defining agent performance and product quality to turning results into useful workflows and product decisions. The strongest candidates will bring experience building or managing agentic systems, either by directly