Staff Product Manager: Agentic AI Evaluations & Metrics

Twenty; New York (NY); On-site; USD 161,000 - 389,000; Full time

Benefits offered by this job

Health plan

Family leave

Paid holidays

401(k) plan

Job summary

Twenty builds AI-enabled technology for mission-critical cyber operations. We're seeking a Staff Product Manager to help make agentic AI systems measurable, reliable, and continuously improvable.

You’ll partner with AI and engineering teams to shape evaluation capabilities—from defining agent performance and product quality to turning results into useful workflows and product decisions. The strongest candidates will bring meaningful experience building or managing agentic systems, either by

Qualifications

• Strong product experience with AI or agentic systems.
• Experience building or managing agentic systems through product work.
• Ability to translate technical evaluation needs into product requirements in partnership with engineering and AI teams.
• Ability to define or operationalize meaningful measures of agent performance and product quality.

Responsibilities

• Own or support the product roadmap for agentic AI evaluations, translating goals into clear priorities and outcomes.
• Partner with engineering and AI teams to understand evaluation needs, constraints, and workflows.
• Define how agent performance and product quality should be assessed, including measures of success, reliability, and quality.
• Translate evaluation needs into clear product requirements, workflows, and acceptance criteria.
• Help operationalize evaluation workflows and success metrics so teams can assess agent behavior and product changes.
• Use evaluation results to identify gaps, inform product decisions, and improve the agent experience over time.

Job description

Twenty builds AI-enabled technology for mission-critical cyber operations. We're seeking a Staff Product Manager to help make agentic AI systems measurable, reliable, and continuously improvable.

You’ll partner with AI and engineering teams to shape evaluation capabilities—from defining agent performance and product quality to turning results into useful workflows and product decisions. The strongest candidates will bring meaningful experience building or managing agentic systems, either by