AI Solutions Analyst - Remote | AI QA & Insights

Allied Benefit Systems, LLC; Chicago (IL); Remote; USD 105,000 - 125,000; Full time

Job summary

Allied Benefit Systems is seeking an AI Business Analyst to evaluate internal AI solutions and translate findings into actionable feedback for product and engineering teams. The role interfaces with Dynamics CRM, claims adjudication platforms, and document repositories, and involves documenting requirements and test scenarios.

The candidate will collaborate with product owners and SMEs, validate AI behavior, and help define acceptance criteria.

Qualifications

• Bachelor’s degree in a related field or equivalent work experience.
• 8–12 years of experience in business analysis, quality assurance, or systems evaluation in insurance or administrative healthcare.
• Strong analytical mindset and ability to identify subtle patterns in complex systems.

Responsibilities

• Evaluate AI capabilities in core systems to determine functionality and business outcomes.
• Investigate edge cases, anomalies, and failure patterns by reviewing logs and outputs.
• Document findings as user stories and product requirements for engineers and product owners.
• Provide business validation of AI behavior and identify gaps in functionality.
• Develop test scenarios and evaluation frameworks reflecting real-world usage.
• Suggest improvements to AI capabilities based on feedback and business context.
• Support acceptance criteria and success metrics for new AI features.
• Ensure AI solutions are understandable and aligned with operational goals.

Skills

Business Analysis

QA

Systems Evaluation

Analytical Thinking

Collaboration

Communication

Customer Focus

Initiative

Functional Expertise

Education

Bachelor's degree

Tools

Excel

SQL

Power BI

Azure Monitor

Splunk

Postman

Azure DevOps

Freshservice

Job description

Allied Benefit Systems is seeking an AI Business Analyst to evaluate internal AI solutions and translate findings into actionable feedback for product and engineering teams. The role interfaces with Dynamics CRM, claims adjudication platforms, and document repositories, and involves documenting requirements and test scenarios.

The candidate will collaborate with product owners and SMEs, validate AI behavior, and help define acceptance criteria.