Manager - AI Tech Project Management - Remote

Taleo; Eden Prairie (MN); Remote; USD 92,000 - 164,000; Full time

Benefits offered by this job

Comprehensive benefits

Equity stock purchase

401k plan

Job summary

Optum Technology seeks a Senior Technical Product Manager to lead AI Patrol – Evals as a Service, guiding strategy, roadmap and enterprise adoption. You will define evaluation dimensions for LLMs, curate test sets, and implement automated scoring across the enterprise.

You will work remotely from anywhere in the U.S., with in-office requirements for Minneapolis or DC when applicable, and collaborate with engineering, data science, governance, and business leaders.

Qualifications

• Undergraduate degree or 4+ years of equivalent technical product or program management experience.
• 5+ years of experience in technical product or program management owning product roadmaps, intake, prioritization, and writing measurable acceptance criteria.
• 2+ years of experience managing platform or shared-service technical products.
• 2+ years of experience with large language models (LLMs), retrieval-augmented generation (RAG), and agentic architectures.
• 2+ years of experience with AI model and application evaluation methods (eg, gold-set design, red-teaming/adversarial testing, hallucination measurement, or drift monitoring).
• 1+ years of experience using SQL and Python for reading evaluation telemetry, metrics, and dashboards.

Responsibilities

• Own the product strategy, roadmap, intake prioritization, and enterprise adoption of AI Patrol – Evals as a Service.
• Design, develop, and deploy AI-powered evaluation solutions to address complex business challenges with an emphasis on responsible AI use.
• Convert fragmented evaluation practices into one governed, reusable shared service with standard scorecards and release gates.
• Define evaluation dimensions for LLMs and agentic solutions, including accuracy, groundedness, hallucination rate, bias, toxicity, latency, and cost per task.
• Curate gold-standard and adversarial test sets, set thresholds for automated LLM-as-judge and human-in-the-loop scoring, and prioritize key platform features.
• Leverage enterprise-approved AI tools to streamline workflows, automate tasks, analyze requirements, summarize telemetry, and produce evaluation reporting.
• Evaluate emerging trends in LLMOps and model evaluation to inform solution design, benchmarking standards, and strategic platform innovation.

Skills

Platform products

Roadmaps

AI governance

LLMs

SQL

Python

Education

Bachelor's degree in CS/engineering

Tools

AWS

Azure

GCP

CI/CD

Job description

Improve the lives of others while Caring. Connecting. Growing together.

Manager - AI Tech Project Management - Remote - 2386989

Optum Tech is a global leader in health care innovation. Our teams develop cutting-edge solutions that help people live healthier lives and help make the health system work better for everyone. From advanced data analytics and AI to cybersecurity, we use innovative approaches to solve some of health care’s most complex challenges. Your contributions here have the potential to change lives. Ready to build the next breakthrough? Join us to start Caring. Connecting. Growing together.

Join Optum Technology to lead the product strategy, roadmap, and enterprise adoption of AI Patrol – Evals as a Service. This critical shared capability gives engineering and product teams a standard way to measure the quality, safety, and cost of AI solutions before and after release. In this role, you will convert fragmented, team-by-team evaluation practices into a governed, reusable service with defined intake, standard scorecards, and automated release gates. Working directly inside the AI stack, you will define evaluation dimensions for large language models and agentic architectures, curate gold-standard and adversarial test sets, and implement automated LLM-as-judge scoring to accelerate the path from prototype to production release across the enterprise.

You’ll enjoy the flexibility to work remotely * from anywhere within the U.S. as you take on some tough challenges. For all hires in the Minneapolis or Washington, D.C. area, you will be required to work in the office a minimum of four days per week.

Primary Responsibilities:

• Own the product strategy, roadmap, intake prioritization, and enterprise adoption of AI Patrol – Evals as a Service
• Design, develop, and deploy AI-powered evaluation solutions to address complex business challenges with an emphasis on responsible AI use
• Convert fragmented evaluation practices into one governed, reusable shared service with standard scorecards and release gates
• Define evaluation dimensions for large language model (LLM) and agentic solutions, including accuracy, groundedness, hallucination rate, bias, toxicity, latency, and cost per task
• Curate gold-standard and adversarial test sets, set thresholds for automated LLM-as-judge and human-in-the-loop scoring, and prioritize key platform features
• Leverage enterprise-approved AI tools to streamline workflows, automate tasks, analyze requirements, summarize telemetry, and produce evaluation reporting
• Evaluate emerging trends in LLMOps and model evaluation to inform solution design, benchmarking standards, and strategic platform innovation

You’ll be rewarded and recognized for your performance in an environment that will challenge you and give you clear direction on what it takes to succeed in your role as well as provide development for other roles you may be interested in.

Required Qualifications:

• Undergraduate degree or 4+ years of equivalent technical product or program management experience
• 5+ years of experience in technical product or program management owning product roadmaps, intake, prioritization, and writing measurable acceptance criteria
• 2+ years of experience managing platform or shared-service technical products
• 2+ years of experience or working fluency with large language models (LLMs), retrieval-augmented generation (RAG), and agentic architectures
• 2+ years of experience with AI model and application evaluation methods (eg, gold-set design, red-teaming/adversarial testing, hallucination measurement, or drift monitoring)
• 1+ years of experience using SQL and Python for reading evaluation telemetry, metrics, and dashboards

Preferred Qualifications:

• Undergraduate degree in Computer Science, Data Science, Engineering, or a related technical field
• Hands-on experience with MLOps/LLMOps tools and CI/CD release gating concepts
• Experience with cloud AI platform services (AWS, Azure, GCP)
• Familiarity with healthcare data handling, including PHI and HIPAA compliance obligations
• Demonstrated ability to influence without authority across engineering, data science, governance, and business leaders
• Executive-level communication skills with experience presenting standardized evidence for AI governance, risk, and compliance stakeholders

*All employees working remotely will be required to adhere to UnitedHealth Group’s Telecommuter Policy

Pay is based on several factors including but not limited to local labor markets, education, work experience, certifications, etc. In addition to your salary, we offer benefits such as, a comprehensive benefits package, incentive and recognition programs, equity stock purchase and 401k contribution (all benefits are subject to eligibility requirements). No matter where or when you begin a career with us, you’ll find a far-reaching choice of benefits and incentives. The salary for this role will range from $91,700 - $163,700 annually based on full-time employment. We comply with all minimum wage laws as applicable.

Application Deadline: This will be posted for a minimum of 2 business days or until a sufficient candidate pool has been collected. Job posting may come down early due to volume of applicants.

At UnitedHealth Group, our mission is to help people live healthier lives and make the health system work better for everyone. We believe everyone–of every race, gender, sexuality, age, location and income–deserves the opportunity to live their healthiest life. Today, however, there are still far too many barriers to good health which are disproportionately experienced by people of color, historically marginalized groups and those with lower incomes. We are committed to mitigating our impact on the environment and enabling and delivering equitable care that addresses health disparities and improves health outcomes — an enterprise priority reflected in our mission.

UnitedHealth Group is an Equal Employment Opportunity employer under applicable law and qualified applicants will receive consideration for employment without regard to race, national origin, religion, age, color, sex, sexual orientation, gender identity, disability, or protected veteran status, or any other characteristic protected by local, state, or federal laws, rules, or regulations.

UnitedHealth Group is a drug - free workplace. Candidates are required to pass a drug test before beginning employment.

UnitedHealth Group is committed to working with and providing reasonable accommodations to individuals with physical and mental disabilities. If you need special assistance or accommodation for any part of the application process, please call 1-866-566-8715 to be connected to Recruitment Services. Recruitment Services hours of operation are 7 a.m. to 7 p.m. CT, Monday through Friday.

UnitedHealth Group is a registered service mark of UnitedHealth Group, Inc. The UnitedHealth Group name with the dimensional logo, as well as the dimensional logo alone, are both service marks for the UnitedHealth Group, Inc.

Diversity creates a healthier atmosphere: UnitedHealth Group is an Equal Employment Opportunity/Affirmative Action employer and all qualified applicants will receive consideration for employment without regard to race, color, religion, sex, age, national origin, protected veteran status, disability status, sexual orientation, gender identity or expression, marital status, genetic information, or any other characteristic protected by law.

UnitedHealth Group is a drug-free workplace. Candidates are required to pass a drug test before beginning employment.