AI Quality & Reliability Lead

Refinitiv; Eagan (MN); Hybrid; USD 91,000 - 169,000; Full time

Benefits offered by this job

Hybrid work model

Mental health days

Tuition reimbursement

Retirement plan

Volunteer days

Job summary

Thomson Reuters in the United States is seeking an AI Reliability Manager to ensure the accuracy, trust, and continuous improvement of agentic AI within CLEAR, our investigative platform used by law enforcement, financial crimes teams, and government agencies. You will own the evaluation data, extend gold datasets, and drive release readiness through rigorous testing and cross-functional collaboration.

Ideal candidates will have hands-on experience with generative AI tools, strong written

Qualifications

• Three or more years, or equivalent experience, in work requiring careful judgment about quality, such as research, analysis, editorial, audit, quality assurance, or investigative work
• Experience applying consistent standards to open-ended work where reasonable people can disagree about what "good" looks like
• Hands-on use of generative AI tools, with enough curiosity to have noticed how they fail, including confident answers that aren't supported and sources that don't say what the AI claims
• Strong written communication, including explaining technical findings to non-technical audiences

Responsibilities

• Own and extend the gold datasets, annotation guidelines, and scoring rubrics used to evaluate agentic AI output for accuracy, completeness, and appropriate sourcing
• Execute evaluation cycles, scoring AI responses against established guidelines, maintaining consistency across annotators, and flagging where output falls short
• Analyze evaluation results to identify failure patterns, quantify their impact, and recommend where engineering and data science effort should be focused
• Test and validate AI features prior to release, and provide a clear, evidence-based point of view on release readiness
• Train and support additional cross functional annotator resources, including globally distributed contributors who participate in evaluation cycles but are not day-to-day experts, ensuring they apply guidelines consistently
• Serve as a day-to-day product resource for the sales channel, answering questions about product behaviour, capabilities, and known limitations, and keeping them current on the status of open issues
• Own intake of reported issues across the CLEAR application, from AI output quality concerns to general product defects; reproduce issues, determine root cause category, and document them precisely
• Open and manage engineering and labs tickets, and drive them to resolution alongside product, engineering, and data science partners
• Track recurring defect themes over time and surface them to product leadership as systemic issues rather than one-off tickets
• Partner with product management, engineering, applied research, and go-to-market teams to translate quality findings into roadmap and prioritization decisions
• Build an understanding of customer pain points and goals, and help identify new ways agentic AI can be applied to investigative workflows

Skills

Quality judgment

Consistency standards

Generative AI tools

Written communication

Job description

Thomson Reuters in the United States is seeking an AI Reliability Manager to ensure the accuracy, trust, and continuous improvement of agentic AI within CLEAR, our investigative platform used by law enforcement, financial crimes teams, and government agencies. You will own the evaluation data, extend gold datasets, and drive release readiness through rigorous testing and cross-functional collaboration.

Ideal candidates will have hands-on experience with generative AI tools, strong written