AI Product Leader - Enterprise Evaluation & Governance

Taleo; Eden Prairie (MN); Remote; USD 92,000 - 164,000; Full time

Benefits offered by this job

Comprehensive benefits

Equity stock purchase

401k plan

Job summary

Optum Technology seeks a Senior Technical Product Manager to lead AI Patrol – Evals as a Service, guiding strategy, roadmap and enterprise adoption. You will define evaluation dimensions for LLMs, curate test sets, and implement automated scoring across the enterprise.

You will work remotely from anywhere in the U.S., with in-office requirements for Minneapolis or DC when applicable, and collaborate with engineering, data science, governance, and business leaders.

Qualifications

• Undergraduate degree or 4+ years of equivalent technical product or program management experience.
• 5+ years of experience in technical product or program management owning product roadmaps, intake, prioritization, and writing measurable acceptance criteria.
• 2+ years of experience managing platform or shared-service technical products.
• 2+ years of experience with large language models (LLMs), retrieval-augmented generation (RAG), and agentic architectures.
• 2+ years of experience with AI model and application evaluation methods (eg, gold-set design, red-teaming/adversarial testing, hallucination measurement, or drift monitoring).
• 1+ years of experience using SQL and Python for reading evaluation telemetry, metrics, and dashboards.

Responsibilities

• Own the product strategy, roadmap, intake prioritization, and enterprise adoption of AI Patrol – Evals as a Service.
• Design, develop, and deploy AI-powered evaluation solutions to address complex business challenges with an emphasis on responsible AI use.
• Convert fragmented evaluation practices into one governed, reusable shared service with standard scorecards and release gates.
• Define evaluation dimensions for LLMs and agentic solutions, including accuracy, groundedness, hallucination rate, bias, toxicity, latency, and cost per task.
• Curate gold-standard and adversarial test sets, set thresholds for automated LLM-as-judge and human-in-the-loop scoring, and prioritize key platform features.
• Leverage enterprise-approved AI tools to streamline workflows, automate tasks, analyze requirements, summarize telemetry, and produce evaluation reporting.
• Evaluate emerging trends in LLMOps and model evaluation to inform solution design, benchmarking standards, and strategic platform innovation.

Skills

Platform products

Roadmaps

AI governance

LLMs

SQL

Python

Education

Bachelor's degree in CS/engineering

Tools

AWS

Azure

GCP

CI/CD

Job description

Optum Technology seeks a Senior Technical Product Manager to lead AI Patrol – Evals as a Service, guiding strategy, roadmap and enterprise adoption. You will define evaluation dimensions for LLMs, curate test sets, and implement automated scoring across the enterprise.

You will work remotely from anywhere in the U.S., with in-office requirements for Minneapolis or DC when applicable, and collaborate with engineering, data science, governance, and business leaders.