AI Performance & Production Reliability Manager

Devoted Health Services, Inc; Massachusetts; On-site; USD 110,000 - 172,000; Full time

Benefits offered by this job

Employer sponsored health, dental and vision plan

Generous paid time off

$100 monthly mobile or internet stipend

Stock options for all employees

Bonus eligibility for all roles

Parental leave program

401K program

Job summary

Devoted Health Services, Inc is seeking a Senior Manager, AI Performance & Operations to own and improve AI workflows, ensuring high-quality performance and operational outcomes.

This role involves maintaining oversight of major workflows, leading incident responses, and driving continuous optimization. Candidates should possess strong expertise in AI systems, strategic communication skills, and be ready to take ownership of outcomes while promoting a diverse workforce.

Qualifications

• Strong production experience with LLMs, RAG, voice AI, or equivalent ML systems.
• Experience designing or operating evaluation systems and rubrics.
• Ability to diagnose various AI-related issues effectively.
• Comfort with experimentation and iterative testing approaches.
• Excellent cross-functional communication skills.
• Strong personal accountability and result-oriented mindset.

Responsibilities

• Own the health of major voice and text AI workflows post-launch.
• Maintain an up-to-date view of all major workflows' statuses.
• Monitor performance indicators and oversee production dashboards.
• Lead responses for AI regressions and drive problem resolution.
• Run structured cycles for continuous improvement and optimization.
• Design evaluation systems for production and pre-production.
• Translate technical changes for leadership and stakeholders.

Skills

Applied AI Expertise

Evaluation Systems

Diagnostic Acumen

Optimization Mindset

Strategic Communication

Outcome Ownership

Education

Relevant degree or equivalent experience

Job description

Devoted Health Services, Inc is seeking a Senior Manager, AI Performance & Operations to own and improve AI workflows, ensuring high-quality performance and operational outcomes.

This role involves maintaining oversight of major workflows, leading incident responses, and driving continuous optimization. Candidates should possess strong expertise in AI systems, strategic communication skills, and be ready to take ownership of outcomes while promoting a diverse workforce.