Remote AI Business Quality Analyst

Allied Benefit Systems; Chicago (IL); Remote; USD 90,000 - 120,000; Full time

Benefits offered by this job

Medical Insurance

Dental Insurance

Vision Insurance

Life and Disability Insurance

Generous Paid Time Off

Tuition Reimbursement

Technology Stipend

Job summary

Allied Benefit Systems is seeking an AI Business Analyst to evaluate whether our internal AI solutions are performing as intended and to translate findings into actionable feedback for product and engineering teams.

You will review AI capabilities across core systems, identify edge cases, collaborate with technical owners, and help translate insights into user stories and product requirements, while supporting testing and validation efforts.

Qualifications

• Bachelor’s degree in related field or equivalent work experience required.
• 8–12 years of experience in business analysis, quality assurance, or systems evaluation roles, ideally within the insurance or administrative healthcare industry
• Strong analytical mindset with the ability to identify subtle patterns, inconsistencies, or edge cases in complex systems
• Comfortable working with structured and unstructured data, including reviewing logs, outputs, and raw files to trace issues or validate behavior
• Experience writing clear, structured documentation such as user stories, test cases, and issue reports
• Familiarity with AI/ML systems and their evaluation is a plus, but not required
• Excellent communication skills and the ability to collaborate across technical and non-technical teams
• Familiarity with tools for data inspection and analysis (e.g., Excel, SQL, Power BI)
• Experience with log analysis and debugging tools (e.g., Azure Monitor, Splunk)
• Exposure to API testing tools (e.g., Postman) and basic scripting or query languages is helpful
• Agile collaboration tools (e.g., Freshservice, Azure DevOps, or similar) for tracking issues and documenting requirements
• Play a key role in Allied’s AI delivery team, helping ensure that AI capabilities are not only technically sound but also aligned with business needs and user expectations

Responsibilities

• Evaluate AI capabilities integrated into core systems (e.g., Dynamics CRM, claims adjudication platform, document repositories) to determine whether they are functioning as intended and delivering expected business outcomes
• Investigate edge cases, anomalies, and failure patterns by reviewing raw data, logs, and system outputs
• Collaborate with technical product owners and SMEs to document what is and isn’t working, and help translate findings into user stories and product requirements
• Participate in model and capability reviews, providing business-level validation of AI behavior and identifying gaps in functionality or alignment with intended use
• Contribute to the development of test scenarios and evaluation frameworks that reflect real-world usage and business expectations
• Recommend improvements or refinements to AI capabilities based on observed behavior, user feedback, and business context
• Support the AI Product Owner and QA team in defining acceptance criteria and success metrics for new AI features
• Help ensure that AI solutions are understandable, trustworthy, and aligned with Allied’s operational goals
• Other duties as assigned

Skills

Analytical Thinking

Communication

Collaboration

Data Analysis

Documentation

SQL

Excel

Education

Bachelor’s degree in related field

Tools

Azure Monitor

Splunk

Postman

Freshservice

Azure DevOps

Power BI

Job description

Allied Benefit Systems is seeking an AI Business Analyst to evaluate whether our internal AI solutions are performing as intended and to translate findings into actionable feedback for product and engineering teams.

You will review AI capabilities across core systems, identify edge cases, collaborate with technical owners, and help translate insights into user stories and product requirements, while supporting testing and validation efforts.