Senior Manager, AI Performance & Operations

Devoted Health Services, Inc; Massachusetts; On-site; USD 110,000 - 172,000; Full time

Benefits offered by this job

Employer sponsored health, dental and vision plan

Generous paid time off

$100 monthly mobile or internet stipend

Stock options for all employees

Bonus eligibility for all roles

Parental leave program

401K program

Job summary

Devoted Health Services, Inc is seeking a Senior Manager, AI Performance & Operations to own and improve AI workflows, ensuring high-quality performance and operational outcomes.

This role involves maintaining oversight of major workflows, leading incident responses, and driving continuous optimization. Candidates should possess strong expertise in AI systems, strategic communication skills, and be ready to take ownership of outcomes while promoting a diverse workforce.

Qualifications

• Strong production experience with LLMs, RAG, voice AI, or equivalent ML systems.
• Experience designing or operating evaluation systems and rubrics.
• Ability to diagnose various AI-related issues effectively.
• Comfort with experimentation and iterative testing approaches.
• Excellent cross-functional communication skills.
• Strong personal accountability and result-oriented mindset.

Responsibilities

• Own the health of major voice and text AI workflows post-launch.
• Maintain an up-to-date view of all major workflows' statuses.
• Monitor performance indicators and oversee production dashboards.
• Lead responses for AI regressions and drive problem resolution.
• Run structured cycles for continuous improvement and optimization.
• Design evaluation systems for production and pre-production.
• Translate technical changes for leadership and stakeholders.

Skills

Applied AI Expertise

Evaluation Systems

Diagnostic Acumen

Optimization Mindset

Strategic Communication

Outcome Ownership

Education

Relevant degree or equivalent experience

Job description

Job Overview

As a Senior Manager, AI Performance & Operations, you will own the production performance and continuous improvement of Devoted’s core agentic workflows for service and experience so that every general-availability workflow has clear performance thresholds, reliable monitoring, explicit fallback paths, and a measurable improvement plan. This person is the accountable operator for Guide’s highest-volume and highest-risk AI workflows once they are live. They maintain the portfolio view of where each workflow stands, detect degradation early, diagnose the root cause, drive corrective action, and verify that implemented changes improve outcomes. They also serve as the in-house expert for early-stage projects by helping teams define instrumentation, evaluation design, launch criteria, and safe hand off into general availability.

Responsibilities and Impact

• Post-Launch Accountability: Own the health of major voice and text AI workflows, ensuring business, quality, safety, and operational outcomes are met.
• Portfolio Oversight: Maintain a current view of all major workflows, including status, maturity, failure modes, and next actions.
• Performance & Dashboards: Monitor weekly indicators (containment, escalation, complaints) and oversee production dashboards to track behavior independently.
• Diagnosis & Incident Response: Lead response for AI regressions; distinguish between model, prompt, or orchestration issues and drive post-mortems to closure.
• Continuous Improvement & Optimization: Run structured cycles (hypothesis, testing, rollout/rollback) for "mini-launches" like model swaps, prompt changes, and tool updates.
• Recommend evidence-based accelerations, such as model migrations or deeper tool-orchestration changes.
• Scalable Evaluation: Design eval systems using golden datasets, LLM-as-judge workflows, and human review queues for production and pre-production.
• Expert Review: Guide new workflows during incubation to ensure they launch with proper instrumentation, eval coverage, and safety models.
• Executive Communication: Translate technical changes into plain language for leadership, explaining what broke, why it matters, and who is fixing it.

Qualifications

• Applied AI Expertise: Strong production experience with LLMs, RAG, voice AI, agentic orchestration, or equivalent ML systems.
• Evaluation Systems: Demonstrated experience designing or operating rubrics, scoring frameworks, regression tests, and production-sampling workflows.
• Diagnostic Acumen: A clear mental model for distinguishing failure modes, such as model reasoning, tool failure, retrieval issues, and workflow logic.
• Optimization Mindset: Comfort with active experimentation—including prompt/context tuning, retrieval redesign, and safe model migrations—beyond simple issue triage.
• Strategic Communication: Excellent cross-functional judgment and the ability to communicate clearly in ambiguous, high-accountability environments.
• Outcome Ownership: Deep personal accountability with a drive to own results rather than just producing recommendations.
• Preferred Industry Background: Experience in healthcare, regulated workflows, contact center operations, or high-stakes service quality/reliability functions.

Salary

Salary: $110,000 - $172,000 / year. The pay range listed for this position is the range the organization reasonably and in good faith expects to pay for this position at the time of the posting. Once the interview process begins, your talent partner will provide additional information on the compensation for the role, along with additional information on our total rewards package. The actual base salary offered will depend on a variety of factors, including the qualifications of the individual applicant for the position, years of relevant experience, specific and unique skills, level of education attained, certifications or other professional licenses held, and the location in which the applicant lives and/or from which they will be performing the job.

Benefits

• Employer sponsored health, dental and vision plan with low or no premium
• Generous paid time off
• $100 monthly mobile or internet stipend
• Stock options for all employees
• Bonus eligibility for all roles excluding Director and above; Commission eligibility for Sales roles
• Parental leave program
• 401K program
• And more...

Equal Opportunity Employer Statement

Healthcare equality is at the center of Devoted’s mission to treat our members like family. We are committed to a diverse and vibrant workforce. At Devoted Health, we’re on a mission to dramatically improve the health and well-being of older Americans by caring for every person like family. That’s why we’re gathering smart, diverse, and big-hearted people to create a new kind of all-in-one healthcare company — one that combines compassion, health insurance, clinical care, service, and technology - to deliver a complete and integrated healthcare solution that delivers high quality care that everyone would want for someone they love. Founded in 2017, we've grown fast and now serve members across the United States. And we've just started. So join us on this mission! Devoted is an equal opportunity employer. We are committed to a safe and supportive work environment in which all employees have the opportunity to participate and contribute to the success of the business. We value diversity and collaboration. Individuals are respected for their skills, experience, and unique perspectives. This commitment is embodied in Devoted’s Code of Conduct, our company values and the way we do business. As an Equal Opportunity Employer, the Company does not discriminate on the basis of race, color, religion, sex, pregnancy status, marital status, national origin, disability, age, sexual orientation, veteran status, genetic information, gender identity, gender expression, or any other factor prohibited by law. Our management team is dedicated to this policy with respect to recruitment, hiring, placement, promotion, transfer, training, compensation, benefits, employee activities and general treatment during employment. We have been made aware of instances of fraudulent job postings and/or fraudulent recruiting activity by individuals purporting to represent Devoted Health. These fraudulent schemes often seek monetary contributions or payments from job seekers (such as for “start up costs” or “equipment”), or seek to collect sensitive personal information. These job postings and offers are NOT authorized by Devoted Health and Devoted is not responsible for fraudulent offers, personal information that you may have disclosed, or payments made to third parties purporting to represent Devoted. We have reported this matter and are cooperating with law enforcement agencies. Devoted Health will never ask for financial commitment or contribution from a candidate at any stage of the recruitment process.