AI Reliability Manager

Reuters; Eagan, Northern (MN, KY); Hybrid; USD 91,000 - 169,000; Full time

Benefits offered by this job

Hybrid work model

Mental Health Days

Tuition reimbursement

Employee assistance program

Job summary

Refinitiv is seeking an AI Reliability Manager to ensure the integrity, trust, and continuous improvement of agentic AI within the CLEAR platform used by investigators and agencies.

You will own evaluation programs, extend gold datasets and scoring rubrics, and translate findings into actionable engineering work while supporting release readiness with cross-functional partners.

Qualifications

• Three or more years of work requiring careful judgment about quality in research, analysis, editorial, audit, or investigative work.
• Experience applying consistent standards to open-ended work with subjective quality definitions.
• Hands-on use of generative AI tools and ability to cite sources.

Responsibilities

• Own and extend gold datasets, annotation guidelines, and scoring rubrics for AI output quality.
• Execute evaluation cycles, score AI responses, and flag shortfalls across annotators.
• Reproduce issues, determine root cause, and document them precisely.
• Collaborate with product, engineering, and data science to translate findings into roadmap.

Skills

Quality evaluation

AI tooling

Written communication

Tools

Generative AI tools

Job description

**AI Reliability Manager****Job Description:** The Risk & Fraud Product Management team is looking for an AI Reliability Manager to help ensure the agentic AI capabilities in CLEAR, our investigative platform, are accurate, trustworthy, and continuously improving. Our customers include law enforcement, financial crimes compliance teams, corporate fraud investigators, and government agencies who use our products to make consequential decisions. The quality of what our AI produces matters enormously, and this role exists to measure that quality, defend it, and drive it upward.The AI Reliability Manager is an individual contributor who is an analytically rigorous, resourceful problem-solver who is genuinely curious about how agentic AI systems work and how to make them better. You and the reliability team will own the evaluation data and quality signals that tell us whether our AI is performing to standard, translate customer-reported issues into actionable engineering work, and serve as a trusted voice on release readiness. This is a hands-on role for someone who likes digging into messy output, finding the pattern in it, and turning that pattern into tangible improvements.You will join an established evaluation program with existing gold datasets, annotation guidelines, and scoring rubrics already in production use, and you will be responsible for extending and scaling that work as our AI capabilities expand.**Key Responsibilities** **Evaluation & Quality Measurement*** Own and extend the gold datasets, annotation guidelines, and scoring rubrics used to evaluate agentic AI output for accuracy, completeness, and appropriate sourcing.* Execute evaluation cycles, scoring AI responses against established guidelines, maintaining consistency across annotators, and flagging where output falls short.* Analyze evaluation results to identify failure patterns, quantify their impact, and recommend where engineering and data science effort should be focused.* Test and validate AI features prior to release, and provide a clear, evidence-based point of view on release readiness.* Train and support additional cross functional annotator resources, including globally distributed contributors who participate in evaluation cycles but are not day-to-day experts, ensuring they apply guidelines consistently.**Product Support & Issue Resolution*** Serve as a day-to-day product resource for the sales channel, answering questions about product behavior, capabilities, and known limitations, and keeping them current on the status of open issues.* Own intake of reported issues across the CLEAR application, from AI output quality concerns to general product defects; reproduce issues, determine root cause category, and document them precisely.* Open and manage engineering and labs tickets, and drive them to resolution alongside product, engineering, and data science partners.* Track recurring defect themes over time and surface them to product leadership as systemic issues rather than one-off tickets.* Cross-Functional Partnership* Partner with product management, engineering, applied research, and go-to-market teams to translate quality findings into roadmap and prioritization decisions.* Build an understanding of customer pain points and goals, and help identify new ways agentic AI can be applied to investigative workflows.**Required Qualifications*** Three or more years, or equivalent experience, in work requiring careful judgment about quality, such as research, analysis, editorial, audit, quality assurance, or investigative work* Experience applying consistent standards to open-ended work where reasonable people can disagree about what \"good\" looks like* Hands-on use of generative AI tools, with enough curiosity to have noticed how they fail, including confident answers that aren't supported and sources that don't say what the AI claims* Strong written communication, including explaining technical findings to non-technical audiences**Preferred Qualifications:*** Experience in one of our primary customer segments: law enforcement or criminal investigations, AML/BSA or financial crimes compliance, corporate fraud or corporate security, or government program integrity.* Working understanding of how retrieval-based AI systems generate and cite sources, and of common failure modes including hallucination, citation mismatch, and incomplete retrieval.#LI-DS4 **What’s in it For You?*** **Hybrid Work Model:** We’ve adopted a flexible hybrid working environment for our office-based roles while delivering a seamless experience that is digitally and physically connected.* **Flexibility & Work-Life Balance:** Flex My Way is a set of supportive workplace policies designed to help manage personal and professional responsibilities, whether caring for family, giving back to the community, or finding time to refresh and reset. This builds upon our flexible work arrangements, including work from anywhere for up to 8 weeks per year, empowering employees to achieve a better work-life balance.* **Career Development and Growth:** By fostering a culture of continuous learning and skill development, we prepare our talent to tackle tomorrow’s challenges and deliver real-world solutions. Our Grow My Way programming and skills-first approach ensures you have the tools and knowledge to grow, lead, and thrive in an AI-enabled future.* **Industry Competitive Benefits:** We offer comprehensive benefit plans to include flexible vacation, two company-wide Mental Health Days off, access to the Headspace app, retirement savings, tuition reimbursement, employee incentive programs, and resources for mental, physical, and financial wellbeing.* **Culture:** Globally recognized, award-winning reputation for inclusion and belonging, flexibility, work-life balance, and more. We live by our values: Obsess over our Customers, Compete to Win, Challenge (Y)our Thinking, Act Fast / Learn Fast, and Stronger Together.* **Social Impact:** Make an impact in your community with our Social Impact Institute. We offer employees two paid volunteer days off annually and opportunities to get involved with pro-bono consulting projects and Environmental, Social, and Governance (ESG) initiatives.* **Making a Real-World Impact:** We are one of the few companies globally that helps its customers pursue justice, truth, and transparency. Together, with the professionals and institutions we serve, we help uphold the rule of law, turn the wheels of commerce, catch bad actors, report the facts, and provide trusted, unbiased information to people all over the world. In the United States, Thomson Reuters offers a comprehensive benefits package to our employees. Our benefit package includes market competitive health, dental, vision, disability, and life insurance programs, as well as a competitive 401k plan with company match. In addition, Thomson Reuters offers market leading work life benefits with competitive vacation, sick and safe paid time off, paid holidays (including two company mental health days off), parental leave, sabbatical leave. These benefits meet or exceeds the requirements of paid time off in accordance with any applicable state or municipal laws. Finally, Thomson Reuters offers the following additional benefits: optional hospital, accident and sickness insurance paid 100% by the employee; optional life and AD&D insurance paid 100% by the employee; Flexible Spending and Health Savings Accounts; fitness reimbursement; access to Employee Assistance Program; Group Legal Identity Theft Protection benefit paid 100% by employee; access to 529 Plan; commuter benefits; Adoption & Surrogacy Assistance; Tuition Reimbursement; and access to Employee Stock Purchase Plan.Thomson Reuters complies with local laws that require upfront disclosure of the expected pay range for a position. The base compensation range varies across locations.For any eligible US locations, unless otherwise noted, the base compensation range for this role is $91,000 USD - $169,000 USD.Base pay is positioned within the range based on several factors including an individual’s knowledge, skills and experience with consideration given to internal equity. Base pay is one part of a comprehensive Total Reward program which also includes flexible and supportive benefits and other wellbeing programs.This role may also be eligible for an Annual Bonus based on a combination of enterprise and individual performance.