Enterprise AI Operations Engineer — Quality, Governance & ROI

Trissential, LLC; Minneapolis (MN); Hybrid; USD 187,000 - 208,000; Full time

Benefits offered by this job

Competitive compensation

Benefits & healthcare

Paid holidays & PTO

Career development

Strong team culture

Job summary

Trissential, LLC is seeking a Customer AI Operations Engineer to own the operational success of enterprise AI agents, virtual assistants, and generative AI solutions in production. This role partners with product, engineering, operations, and business stakeholders to influence AI quality, governance, adoption, and business outcomes.

You will monitor AI quality and compliance, develop and maintain operational scorecards, review transcripts, investigate issues, coordinate incident responses, and

Qualifications

• 5+ years of experience in technology operations, analytics, product operations, or related fields.
• Minimum 2 years supporting AI agents or generative AI in production.
• Experience evaluating AI responses with transcript reviews and scorecards.
• Strong operational analytics experience with dashboards, KPIs, and performance reporting.
• Knowledge of retrieval operations, search, and knowledge management.
• Experience investigating production incidents and root-cause analysis.
• Understanding of Responsible AI principles, governance, privacy, and compliance.
• Excellent communication and ability to explain AI quality to stakeholders.
• Strong documentation and continuous-improvement skills.

Responsibilities

• Own the operational performance of AI-powered solutions after implementation and hypercare.
• Monitor AI quality, response accuracy, relevance, compliance, adoption, and outcomes.
• Develop and maintain operational scorecards covering containment, satisfaction, escalation rates, retrieval effectiveness, response quality, latency, and utilization.
• Review transcripts and AI conversations to evaluate accuracy, grounding, compliance, and overall response quality.
• Investigate hallucinations, retrieval failures, misrouted interactions, prompt issues, knowledge gaps, and workflow breakdowns.
• Manage and optimize Knowledge Hub performance through content gap analysis, search quality evaluations, source validation, and freshness monitoring.
• Support Responsible AI monitoring activities including privacy validation, governance reporting, audit readiness, bias reviews, and guardrail compliance.
• Coordinate incident investigations and identify root causes across business and technical teams.
• Translate operational findings into prioritized backlog items for Product Owners and AI Engineering teams.
• Measure and report value realization, including productivity gains, self-service adoption, customer outcomes, utilization metrics, and ROI.
• Develop operational procedures, governance standards, and knowledge-transfer documentation to support long-term sustainability.

Skills

Technology operations

Analytics

Product operations

Quality engineering

Knowledge management

Conversational AI

Communication skills

Education

Bachelor's degree in Information Technology or related field

Certifications in Microsoft, Cloud Technologies, AI, IT Service Management, Analytics

Tools

Microsoft Copilot Studio

Salesforce Agentforce

Jira

AI observability tools

Enterprise chatbot platforms

Job description

Trissential, LLC is seeking a Customer AI Operations Engineer to own the operational success of enterprise AI agents, virtual assistants, and generative AI solutions in production. This role partners with product, engineering, operations, and business stakeholders to influence AI quality, governance, adoption, and business outcomes.

You will monitor AI quality and compliance, develop and maintain operational scorecards, review transcripts, investigate issues, coordinate incident responses, and