AI Operations & Quality Engineer

Jobtailor

Colorado

On-site

USD 110,000 - 150,000

Full time

3 days ago

Be an early applicant

Get more replies from employers

Send a job-specific resume in minutes.

Job summary

Jobtailor is seeking an experienced AI Operations/Product Operations professional to own the end-to-end performance of AI-powered solutions and oversee quality, compliance, and value realization. You will develop dashboards, review transcripts, and coordinate incident investigations while translating findings into actionable backlog items for cross-functional teams.

You will collaborate with Product Owners and AI Engineering to drive continuous improvement, governance, privacy, and risk

Qualifications

5+ years of experience in technology operations, analytics, product operations, or related field.
At least 2 years supporting AI agents or generative AI in production.
Experience evaluating AI responses via transcripts, grading, and scorecards.
Strong operational analytics with dashboards, KPIs and trend analysis.
Knowledge of retrieval operations, content-gap analysis and search quality.

Responsibilities

Own operational performance of AI-powered solutions after implementation.
Monitor AI quality, response accuracy, relevance, compliance, adoption, and outcomes.
Develop and maintain operational scorecards covering various metrics.
Review transcripts and AI conversations for accuracy and compliance.
Investigate hallucinations, retrieval failures, prompt issues, and knowledge gaps.
Manage Knowledge Hub performance through gap analysis and freshness checks.
Support Responsible AI monitoring including privacy validation and governance.
Coordinate incident investigations and drive root-cause analysis.
Translate findings into backlog items for Product Owners and AI Engineers.
Measure value realization including ROI and self-service adoption.
Develop governance standards and knowledge-transfer docs for sustainability.