Remote AI Product Lead: Evaluation & Governance

Optum; Eden Prairie (MN); Hybrid; USD 92.000 - 164.000; Vollzeit

Benefits dieser Stelle

Remote work

Comprehensive benefits

Zusammenfassung

Optum Technology seeks a seasoned Technical Product/Program Manager to own the strategy, roadmap, and enterprise adoption of AI Patrol - Evals as a Service. You will forge a governed, reusable evaluation service with standardized scorecards and automated release gates, spanning LLM and agentic architectures.

This remote-friendly role requires in-office presence four days a week for Minneapolis or DC-area hires, collaborating with engineering, data science, governance, and business leaders to

Qualifikationen

• Undergraduate degree or 4+ years of equivalent PM experience.
• 5+ years owning product roadmaps, intake, prioritization, and writing measurable acceptance criteria.
• 2+ years managing platform or shared-service technical products.
• 2+ years working with LLMs, RAG, and agentic architectures.
• 2+ years evaluating AI models and telemetry with SQL/Python.

Aufgaben

• Own the product strategy, roadmap, intake prioritization, and enterprise adoption of AI Patrol - Evals as a Service.
• Design, develop, and deploy AI-powered evaluation solutions with responsible AI focus.
• Convert fragmented evaluation practices into a governed, reusable shared service with scorecards and release gates.
• Define evaluation dimensions for LLMs and agentic solutions (accuracy, groundedness, bias, etc.).
• Curate gold-standard/adversarial test sets and set thresholds for automated scoring.
• Leverage enterprise AI tools to streamline workflows and produce evaluation reporting.
• Evaluate trends in LLMOps and model evaluation to inform platform design and governance.

Kenntnisse

Product management

Roadmaps

SQL

Python

LLMs

AI evaluation methods

RAG

Stakeholder influence

Ausbildung

Bachelor's degree in CS/Eng/Data Science

Tools

MLOps/LLMOps tools

CI/CD release gates

Cloud AI platforms (AWS/Azure/GCP)

Jobbeschreibung

Optum Technology seeks a seasoned Technical Product/Program Manager to own the strategy, roadmap, and enterprise adoption of AI Patrol - Evals as a Service. You will forge a governed, reusable evaluation service with standardized scorecards and automated release gates, spanning LLM and agentic architectures.

This remote-friendly role requires in-office presence four days a week for Minneapolis or DC-area hires, collaborating with engineering, data science, governance, and business leaders to