AI Business Analyst

Allied Benefit Systems, LLC; Chicago (IL); On-site; USD 105,000 - 125,000; Full time

Job summary

Allied Benefit Systems is seeking an AI Business Analyst to evaluate internal AI solutions and translate findings into actionable feedback for product and engineering teams. The role interfaces with Dynamics CRM, claims adjudication platforms, and document repositories, and involves documenting requirements and test scenarios.

The candidate will collaborate with product owners and SMEs, validate AI behavior, and help define acceptance criteria.

Qualifications

• Bachelor’s degree in a related field or equivalent work experience.
• 8–12 years of experience in business analysis, quality assurance, or systems evaluation in insurance or administrative healthcare.
• Strong analytical mindset and ability to identify subtle patterns in complex systems.

Responsibilities

• Evaluate AI capabilities in core systems to determine functionality and business outcomes.
• Investigate edge cases, anomalies, and failure patterns by reviewing logs and outputs.
• Document findings as user stories and product requirements for engineers and product owners.
• Provide business validation of AI behavior and identify gaps in functionality.
• Develop test scenarios and evaluation frameworks reflecting real-world usage.
• Suggest improvements to AI capabilities based on feedback and business context.
• Support acceptance criteria and success metrics for new AI features.
• Ensure AI solutions are understandable and aligned with operational goals.

Skills

Business Analysis

QA

Systems Evaluation

Analytical Thinking

Collaboration

Communication

Customer Focus

Initiative

Functional Expertise

Education

Bachelor's degree

Tools

Excel

SQL

Power BI

Azure Monitor

Splunk

Postman

Azure DevOps

Freshservice

Job description

# AI Business AnalystFully Remote • Allied Benefit Systems - CHICAGO, IL 60606## OverviewSalary Range$105,000.00 - $125,000.00 SalaryPosition TypeFull TimeCategoryOperations## Description**POSITION SUMMARY**The AI Business Analyst will be responsible for evaluating whether Allied’s internal AI solutions are functioning as intended, identifying where they succeed and where they fall short, and translating those findings into clear, actionable feedback for product and engineering teams.**ESSENTIAL FUNCTIONS*** Evaluate AI capabilities integrated into core systems (e.g., Dynamics CRM, claims adjudication platform, document repositories) to determine whether they are functioning as intended and delivering expected business outcomes* Investigate edge cases, anomalies, and failure patterns by reviewing raw data, logs, and system outputs* Collaborate with technical product owners and SMEs to document what is and isn’t working, and help translate findings into user stories and product requirements* Participate in model and capability reviews, providing business-level validation of AI behavior and identifying gaps in functionality or alignment with intended use* Contribute to the development of test scenarios and evaluation frameworks that reflect real-world usage and business expectations* Recommend improvements or refinements to AI capabilities based on observed behavior, user feedback, and business context* Support the AI Product Owner and QA team in defining acceptance criteria and success metrics for new AI features* Help ensure that AI solutions are understandable, trustworthy, and aligned with Allied’s operational goals* Other duties as assigned**EDUCATION*** Bachelor’s degree in related field or equivalent work experience required.**EXPERIENCE AND SKILLS*** 8–12 years of experience in business analysis, quality assurance, or systems evaluation roles, ideally within the insurance or administrative healthcare industry* Deep subject matter expertise in claims, health call centers, provider operations, or health plan administration* Strong analytical mindset with the ability to identify subtle patterns, inconsistencies, or edge cases in complex systems* Comfortable working with structured and unstructured data, including reviewing logs, outputs, and raw files to trace issues or validate behavior* Experience writing clear, structured documentation such as user stories, test cases, and issue reports* Familiarity with AI/ML systems and their evaluation is a plus, but not required* Excellent communication skills and the ability to collaborate across technical and non-technical teams* Familiarity with tools for data inspection and analysis (e.g., Excel, SQL, Power BI)* Experience with log analysis and debugging tools (e.g., Azure Monitor, Splunk)* Exposure to API testing tools (e.g., Postman) and basic scripting or query languages is helpful* Agile collaboration tools (e.g., Freshservice, Azure DevOps, or similar) for tracking issues and documenting requirements* Play a key role in Allied’s AI delivery team, helping ensure that AI capabilities are not only technically sound but also aligned with business needs and user expectations**POSITION COMPETENCIES*** Accountability* Analytical Thinking* Collaboration* Communication* Customer Focus* Functional Expertise* Initiative**PHYSICAL DEMANDS*** This is a standard desk role requiring extended sitting and computer work.**WORK ENVIRONMENT**Remote Here at Allied, we believe that great talent can thrive from anywhere. Our remote friendly culture offers flexibility and the comfort of working from home; while also ensuring you are set up for success. To support a smooth and efficient remote work experience, the internet connection must be obtained through a cable broadband or fiber optic internet service provider with speeds of at least 100Mbps download/25Mbps upload. Reliable internet service is essential for staying connected and productive.The company has reviewed this job description to ensure that essential functions and basic duties have been included. It is not intended to be construed as an exhaustive list of all functions, responsibilities, skills, and abilities. Additional functions and requirements may be assigned by supervisors as deemed appropriate.Compensation is not limited to base salary. Allied values our Total Rewards, and offers a competitive Benefit Package including, but not limited to, Medical, Dental, Vision, Life and Disability Insurance, Generous Paid Time Off, Tuition Reimbursement, EAP, and a Technology Stipend.Allied reserves the right to amend, change, alter, and revise, pay ranges and benefits offerings at any time. All applicants acknowledge that by applying to the position you understand that the specific pay range is contingent upon meeting the qualification and requirements of the role, and for the successful completion of the interview selection and process. It is at the Company's discretion to determine what pay is provided to a candidate within the range associated with the role.