Customer AI Operations Engineer

Trissential, LLC; Minneapolis (MN); Hybrid; USD 187,000 - 208,000; Full time

Benefits offered by this job

Competitive compensation

Benefits & healthcare

Paid holidays & PTO

Career development

Strong team culture

Job summary

Trissential, LLC is seeking a Customer AI Operations Engineer to own the operational success of enterprise AI agents, virtual assistants, and generative AI solutions in production. This role partners with product, engineering, operations, and business stakeholders to influence AI quality, governance, adoption, and business outcomes.

You will monitor AI quality and compliance, develop and maintain operational scorecards, review transcripts, investigate issues, coordinate incident responses, and

Qualifications

• 5+ years of experience in technology operations, analytics, product operations, or related fields.
• Minimum 2 years supporting AI agents or generative AI in production.
• Experience evaluating AI responses with transcript reviews and scorecards.
• Strong operational analytics experience with dashboards, KPIs, and performance reporting.
• Knowledge of retrieval operations, search, and knowledge management.
• Experience investigating production incidents and root-cause analysis.
• Understanding of Responsible AI principles, governance, privacy, and compliance.
• Excellent communication and ability to explain AI quality to stakeholders.
• Strong documentation and continuous-improvement skills.

Responsibilities

• Own the operational performance of AI-powered solutions after implementation and hypercare.
• Monitor AI quality, response accuracy, relevance, compliance, adoption, and outcomes.
• Develop and maintain operational scorecards covering containment, satisfaction, escalation rates, retrieval effectiveness, response quality, latency, and utilization.
• Review transcripts and AI conversations to evaluate accuracy, grounding, compliance, and overall response quality.
• Investigate hallucinations, retrieval failures, misrouted interactions, prompt issues, knowledge gaps, and workflow breakdowns.
• Manage and optimize Knowledge Hub performance through content gap analysis, search quality evaluations, source validation, and freshness monitoring.
• Support Responsible AI monitoring activities including privacy validation, governance reporting, audit readiness, bias reviews, and guardrail compliance.
• Coordinate incident investigations and identify root causes across business and technical teams.
• Translate operational findings into prioritized backlog items for Product Owners and AI Engineering teams.
• Measure and report value realization, including productivity gains, self-service adoption, customer outcomes, utilization metrics, and ROI.
• Develop operational procedures, governance standards, and knowledge-transfer documentation to support long-term sustainability.

Skills

Technology operations

Analytics

Product operations

Quality engineering

Knowledge management

Conversational AI

Communication skills

Education

Bachelor's degree in Information Technology or related field

Certifications in Microsoft, Cloud Technologies, AI, IT Service Management, Analytics

Tools

Microsoft Copilot Studio

Salesforce Agentforce

Jira

AI observability tools

Enterprise chatbot platforms

Job description

Overview

Location: Hybrid Preferred (Minneapolis, MN or Denver, CO) | Remote ConsideredEmployment Type: Full-time

Are you passionate about ensuring AI solutions deliver real business value after deployment? Trissential is seeking a Customer AI Operations Engineer to join our client's team and take ownership of the operational success of enterprise AI agents, virtual assistants, and generative AI solutions in production. This is a unique opportunity to influence AI quality, governance, adoption, and business outcomes while partnering with product, engineering, operations, and business stakeholders.

What's in It for You?

• Be the operational owner of cutting-edge AI solutions used across a large enterprise environment
• Drive measurable improvements in AI quality, adoption, customer experience, and ROI
• Collaborate with Product, AI Engineering, Operations, Security, Governance, and business SMEs
• Gain exposure to Responsible AI practices, enterprise governance programs, and AI observability platforms
• Work in a highly visible role where your insights directly influence AI strategy and operational outcomes
• Opportunity to help define long-term AI operational standards and potentially transition into a permanent role

Your Role & Responsibilities

• Own the operational performance of AI-powered solutions after implementation and hypercare
• Monitor AI quality, response accuracy, relevance, compliance, adoption, and business outcomes
• Develop and maintain operational scorecards covering containment, satisfaction, escalation rates, retrieval effectiveness, response quality, latency, and utilization
• Review transcripts and AI conversations to evaluate accuracy, grounding, compliance, and overall response quality
• Investigate hallucinations, retrieval failures, misrouted interactions, prompt issues, knowledge gaps, and workflow breakdowns
• Manage and optimize Knowledge Hub performance through content gap analysis, search quality evaluations, source validation, and freshness monitoring
• Support Responsible AI monitoring activities including privacy validation, governance reporting, audit readiness, bias reviews, and guardrail compliance
• Coordinate incident investigations and identify root causes across business and technical teams
• Translate operational findings into prioritized backlog items for Product Owners and AI Engineering teams
• Measure and report value realization, including productivity gains, self-service adoption, customer outcomes, utilization metrics, and ROI
• Develop operational procedures, governance standards, and knowledge-transfer documentation to support long-term sustainability

Skills & Experience You Should Possess

• 5+ years of experience in technology operations, analytics, product operations, quality engineering, knowledge management, conversational AI, or a related field
• At least 2 years supporting AI agents, conversational AI platforms, virtual assistants, or generative AI solutions in a production environment
• Experience evaluating AI responses through transcript reviews, response grading, hallucination analysis, and quality scorecards
• Strong operational analytics experience with dashboards, KPIs, performance reporting, and trend analysis
• Knowledge of retrieval operations, search effectiveness, knowledge management, source validation, and content-gap analysis
• Experience investigating production incidents and driving root-cause analysis efforts
• Understanding of Responsible AI principles, governance frameworks, privacy considerations, and compliance monitoring
• Excellent communication skills with the ability to explain AI quality issues and business impacts to diverse stakeholders
• Strong documentation, organizational, and continuous-improvement capabilities

Bonus Points If You Have

• Experience supporting AI agents within customer service, customer operations, or contact center environments
• Hands-on experience with Microsoft Copilot Studio, Salesforce Agentforce, enterprise chatbot platforms, or related AI ecosystems
• Experience with AI observability tools, conversation analytics platforms, and evaluation frameworks
• Familiarity with Jira or similar work-management platforms
• Background working within regulated, utility, audit-sensitive, healthcare, financial services, or privacy-focused environments
• Experience supporting enterprise search and retrieval systems

Education & Certifications You Need

• Bachelor's degree in Information Technology, Computer Science, Business Analytics, Data Analytics, Engineering, or a related discipline, or equivalent practical experience
• Relevant certifications in Microsoft, Cloud Technologies, AI, IT Service Management, Product Operations, Analytics, or Data Management are beneficial
• Certifications are valued but do not replace hands-on production AI operations experience

What We Offer

At Trissential, we believe talented professionals should have meaningful opportunities to create measurable impact while continuing to grow their careers. By joining our client's team, you'll help shape the future of enterprise AI operations while working in a collaborative and supportive environment.

• Competitive Compensation – $90-$100 per hour. Final compensation is determined based on skill alignment, years of experience, and fair, market-based rates by geography.
• Comprehensive Benefits for you and your dependents – Extensive Medical Plans, Dental, Vision, HSA with company contribution, company sponsored Life Insurance and Short / Long term disability insurance, a host of voluntary plans including legal, hospital, accident, pet wellness, and 401k with matching
• Paid Holidays and Paid Time Off
• Career Development and Advancement – Expleo is a global company in 29 Countries worldwide with a high focus on development
• Strong Team Culture – Collaborate with experienced professionals who value precision, performance, and teamwork

Please note: This position is open only to candidates authorized to work in the United States.