Senior Product Manager, AI Agent Evaluation & Quality Tools

Amazon Web Services (AWS); Seattle (WA); On-site; USD 151,000 - 205,000; Full time

Benefits offered by this job

Health insurance

RSUs

401(k) matching

Paid time off

Job summary

Amazon Development Center U.S., Inc. is seeking a Senior Product Manager, Technical to define and drive the AI agent evaluation framework within the Core Services AI Foundations team.

You will own end-to-end evaluation tooling, quality scoring, and human-in-the-loop workflows to ensure agent performance before and after deployment. You will partner with engineers and applied scientists to translate research into scalable, self-service features that measure correctness, safety, groundedness, and

Qualifications

• Bachelor's degree required.
• Experience owning/driving roadmap strategy and definition.
• Experience delivering features and making tradeoffs for a product.
• Experience guiding engineering discussions on technology decisions and strategy.

Responsibilities

• Own the product vision, strategy, and roadmap for AI agent evaluation capabilities.
• Write detailed product requirements for evaluation framework APIs and scoring algorithms.
• Partner with applied scientists and engineers to translate research into production features.
• Drive customer discovery to generalize evaluation requirements across architectures and deployment patterns.
• Define success metrics and instrumentation to show product adoption and impact.
• Own go-to-market strategy and positioning against AWS services and third-party tools.
• Coordinate with identity, observability, and data platform teams to integrate signals properly.

Skills

Roadmap strategy ownership

Product feature delivery

Cross-functional collaboration

Technical product management

Stakeholder communication

Education

Bachelor's degree

Tools

Tableau

QlikView

Amazon QuickSight

Job description

Amazon Development Center U.S., Inc. is seeking a Senior Product Manager, Technical to define and drive the AI agent evaluation framework within the Core Services AI Foundations team.

You will own end-to-end evaluation tooling, quality scoring, and human-in-the-loop workflows to ensure agent performance before and after deployment. You will partner with engineers and applied scientists to translate research into scalable, self-service features that measure correctness, safety, groundedness, and