AI Reliability Lead: Build Trustworthy, High-Quality AI

PowerToFly; Eagan (MN); Hybrid; USD 91,000 - 169,000; Full time

Benefits offered by this job

Hybrid work model

Flexible vacation

Mental health days

Job summary

Thomson Reuters in the United States is seeking an AI Reliability Manager to lead evaluation and quality measurement for agentic AI within the CLEAR investigative platform. This hands‑on role develops gold datasets, annotation guidelines, and scoring rubrics, translating findings into actionable engineering work and release readiness.

You will partner with product, engineering, and data science teams to diagnose failure patterns, guide improvements, and support cross‑functional annotators

Qualifications

• Three or more years, or equivalent experience, in work requiring careful judgment about quality, such as research, analysis, editorial, audit, quality assurance, or investigative work
• Experience applying consistent standards to open‑ended work where reasonable people can disagree about what "good" looks like
• Hands‑on use of generative AI tools, with enough curiosity to have noticed how they fail, including confident answers that aren't supported and sources that don't say what the AI claims
• Strong written communication, including explaining technical findings to non‑technical audiences

Responsibilities

• Own and extend the gold datasets, annotation guidelines, and scoring rubrics used to evaluate agentic AI output for accuracy, completeness, and appropriate sourcing
• Execute evaluation cycles, scoring AI responses against established guidelines, maintaining consistency across annotators, and flagging where output falls short
• Analyze evaluation results to identify failure patterns, quantify their impact, and recommend where engineering and data science effort should be focused
• Test and validate AI features prior to release, and provide a clear, evidence-based point of view on release readiness
• Train and support additional cross functional annotator resources, including globally distributed contributors who participate in evaluation cycles but are not day‑to‑day experts, ensuring they apply guidelines consistently
• Serve as a day‑to‑day product resource for the sales channel, answering questions about product behavior, capabilities, and known limitations, and keeping them current on the status of open issues
• Own intake of reported issues across the CLEAR application, from AI output quality concerns to general product defects; reproduce issues, determine root cause category, and document them precisely
• Open and manage engineering and labs tickets, and drive them to resolution alongside product, engineering, and data science partners
• Track recurring defect themes over time and surface them to product leadership as systemic issues rather than one‑off tickets
• Partner with product management, engineering, applied research, and go‑to‑market teams to translate quality findings into roadmap and prioritization decisions
• Build an understanding of customer pain points and goals, and help identify new ways agentic AI can be applied to investigative workflows

Skills

Quality judgment

AI tooling

Written communication

Tools

Generative AI tools

Job description

Thomson Reuters in the United States is seeking an AI Reliability Manager to lead evaluation and quality measurement for agentic AI within the CLEAR investigative platform. This hands‑on role develops gold datasets, annotation guidelines, and scoring rubrics, translating findings into actionable engineering work and release readiness.

You will partner with product, engineering, and data science teams to diagnose failure patterns, guide improvements, and support cross‑functional annotators