Customer AI Operations Engineer

Jobtailor; Colorado; On-site; USD 110,000 - 150,000; Full time

Job summary

Jobtailor is seeking an experienced AI Operations/Product Operations professional to own the end-to-end performance of AI-powered solutions and oversee quality, compliance, and value realization. You will develop dashboards, review transcripts, and coordinate incident investigations while translating findings into actionable backlog items for cross-functional teams.

You will collaborate with Product Owners and AI Engineering to drive continuous improvement, governance, privacy, and risk

Qualifications

• 5+ years of experience in technology operations, analytics, product operations, or related field.
• At least 2 years supporting AI agents or generative AI in production.
• Experience evaluating AI responses via transcripts, grading, and scorecards.
• Strong operational analytics with dashboards, KPIs and trend analysis.
• Knowledge of retrieval operations, content-gap analysis and search quality.

Responsibilities

• Own operational performance of AI-powered solutions after implementation.
• Monitor AI quality, response accuracy, relevance, compliance, adoption, and outcomes.
• Develop and maintain operational scorecards covering various metrics.
• Review transcripts and AI conversations for accuracy and compliance.
• Investigate hallucinations, retrieval failures, prompt issues, and knowledge gaps.
• Manage Knowledge Hub performance through gap analysis and freshness checks.
• Support Responsible AI monitoring including privacy validation and governance.
• Coordinate incident investigations and drive root-cause analysis.
• Translate findings into backlog items for Product Owners and AI Engineers.
• Measure value realization including ROI and self-service adoption.
• Develop governance standards and knowledge-transfer docs for sustainability.

Skills

Operational analytics

Incident investigation

AI quality evaluation

Responsible AI

Communication

Documentation

Continuous improvement

Backlog prioritization

Education

Bachelor's degree in IT/CS/Analytics

Tools

Dashboards

Knowledge Management Systems

AI Platforms

Job description

• Own the operational performance of AI-powered solutions after implementation and hypercare
• Monitor AI quality, response accuracy, relevance, compliance, adoption, and business outcomes
• Develop and maintain operational scorecards covering containment, satisfaction, escalation rates, retrieval effectiveness, response quality, latency, and utilization
• Review transcripts and AI conversations to evaluate accuracy, grounding, compliance, and response quality
• Investigate hallucinations, retrieval failures, misrouted interactions, prompt issues, knowledge gaps, and workflow breakdowns
• Manage and optimize Knowledge Hub performance through content gap analysis, search quality evaluations, source validation, and freshness monitoring
• Support Responsible AI monitoring, including privacy validation, governance reporting, audit readiness, bias reviews, and guardrail compliance
• Coordinate incident investigations and identify root causes across business and technical teams
• Translate operational findings into prioritized backlog items for Product Owners and AI Engineering teams
• Measure and report value realization, including productivity gains, self-service adoption, customer outcomes, utilization metrics, and ROI
• Develop operational procedures, governance standards, and knowledge-transfer documentation for long-term sustainability

Requirements

• 5+ years of experience in technology operations, analytics, product operations, quality engineering, knowledge management, conversational AI, or a related field
• At least 2 years supporting AI agents, conversational AI platforms, virtual assistants, or generative AI solutions in a production environment
• Experience evaluating AI responses through transcript reviews, response grading, hallucination analysis, and quality scorecards
• Strong operational analytics experience with dashboards, KPIs, performance reporting, and trend analysis
• Knowledge of retrieval operations, search effectiveness, knowledge management, source validation, and content-gap analysis
• Experience investigating production incidents and driving root-cause analysis efforts
• Understanding of Responsible AI principles, governance frameworks, privacy considerations, and compliance monitoring
• Excellent communication skills to explain AI quality issues and business impacts to diverse stakeholders
• Strong documentation, organizational, and continuous-improvement capabilities
• Bachelor's degree in Information Technology, Computer Science, Business Analytics, Data Analytics, Engineering, or a related discipline, or equivalent practical experience
• Must be authorized to work in the United States

Core Competencies

Demonstrates expertise in operational performance management of AI-powered solutions, focusing on quality evaluation, compliance, and continuous improvement. Proficient in analytics, incident investigation, and governance standards to ensure effective AI operations and stakeholder communication.

Highest-signal resume keywords

• Operational Performance Management
• AI Quality Evaluation
• Incident Investigation
• Operational Analytics
• Responsible AI Principles

Hard Skills

• Technology Operations
• Quality Engineering
• Conversational AI
• Transcript Review
• Response Grading
• KPI Development
• Root-Cause Analysis
• Content-Gap Analysis
• Performance Reporting
• Trend Analysis

Soft Skills

• Excellent Communication
• Organizational Skills
• Continuous Improvement

Industry Keywords

• Governance Frameworks
• Privacy Considerations
• Compliance Monitoring
• Self-Service Adoption
• Business Outcomes

Tools & Technologies

• AI Platforms
• Dashboards
• Knowledge Management Systems