AI Operations Engineer

Insight Global; Denver (CO); On-site; USD 120,000 - 180,000; Full time

Job summary

Insight Global is seeking an AI Operations Engineer to own post-deployment AI solutions, drive performance, governance, and measurable business impact for a major utility client. This role focuses on operational ownership rather than development.

You will partner with Product, Engineering, Security, and Governance teams to ensure quality, compliance, and adoption of AI solutions in production while delivering executive-ready insights.

Qualifications

• Production AI Operations experience in live production environment.

Responsibilities

• Monitor production AI agents and Knowledge Hub performance using dashboards and operational metrics.

Skills

AI Operations

Quality Evaluation

Knowledge Management

Performance Analytics

Root Cause Analysis

Governance & Compliance

Cross-Functional Leadership

Operational Ownership

Customer Ops

Microsoft Copilot Studio

Salesforce Agentforce

Regulated Environments

Certifications

Tools

Microsoft Copilot Studio

Salesforce Agentforce

Job description

Job Description

An employer is looking for an AI Operations Engineer for one of our top utility clients in the Nation. The AI Operations Engineer is set to serve as the long-term operational owner of production AI solutions. This individual will be responsible for ensuring AI agents deliver accurate, trusted, compliant, and high-quality outcomes while driving adoption, continuous improvement, governance, and measurable business value.

Note: This is not an AI Developer or AI Engineering role. It is an operational ownership role focused on AI performance, quality, governance, and business outcomes after deployment.

Day-to-Day Responsibilities

• Monitor production AI agents and Knowledge Hub performance using dashboards, scorecards, and operational metrics.
• Review AI conversations and transcripts to evaluate response quality, accuracy, grounding, relevance, compliance, and user experience.
• Investigate AI quality issues, hallucinations, retrieval failures, incorrect responses, and escalation trends.
• Perform root cause analysis and partner with Product, Engineering, SMEs, Security, and Governance teams to drive resolution.
• Identify content gaps, stale knowledge, and ineffective search results; coordinate improvements with business stakeholders.
• Prioritize and manage continuous improvement backlogs based on operational findings and business impact.
• Track adoption, self-service success, productivity gains, customer outcomes, and ROI metrics.
• Ensure compliance with Responsible AI, governance, privacy, and audit requirements.
• Deliver operational health reporting, quality insights, and executive-ready recommendations.

We are a company committed to creating diverse and inclusive environments where people can bring their full, authentic selves to work every day. We are an equal opportunity/affirmative action employer that believes everyone matters. Qualified candidates will receive consideration for employment regardless of their race, color, ethnicity, religion, sex (including pregnancy), sexual orientation, gender identity and expression, marital status, national origin, ancestry, genetic factors, age, disability, protected veteran status, military or uniformed service member status, or any other status or characteristic protected by applicable laws, regulations, and ordinances. If you need assistance and/or a reasonable accommodation due to a disability during the application or recruiting process, please send a request to [e-mail]. To learn more about how we collect, keep, and process your private information, please review Insight Global's Workforce Privacy Policy: https://insightglobal.com/workforce-privacy-policy/.

Skills and Requirements

• Production AI Operations Experience: 2+ years supporting AI agents, Copilots, chatbots, conversational AI, or Generative AI solutions in a live production environment after deployment.

• AI Quality & Evaluation: Hands-on experience reviewing AI conversations, grading responses, identifying hallucinations, evaluating response quality, and performing trend analysis.

• Knowledge Management & Search Optimization: Experience evaluating search relevance, retrieval effectiveness, failed searches, content gaps, source quality, and knowledge-base performance.

• AI Performance Analytics: Experience monitoring and analyzing AI performance metrics including accuracy, containment, escalation rates, adoption, satisfaction, task completion, and latency.

• Incident Management & Root Cause Analysis: Proven ability to investigate AI performance issues, identify root causes, and drive corrective actions across product and engineering teams.

• Responsible AI & Governance: Understanding of AI guardrails, privacy, compliance, bias/fairness considerations, governance requirements, and audit readiness.

• Cross-Functional Operations Leadership: Experience partnering with Product, Operations, Engineering, Security, Governance, and business stakeholders to drive AI performance improvements.

• Continuous Improvement & Operational Ownership: Proven experience owning production AI solutions after deployment, translating operational insights into prioritized backlogs, optimization initiatives, and measurable business outcomes. - Customer Service, Contact Center, or Customer Operations experience

• Microsoft Copilot Studio

• Salesforce Agentforce

• Utility, regulated industry, or compliance-heavy environments.

• Microsoft, AI, Cloud, Product Operations, or ITSM certifications.