Artificial Intelligence Quality Engineer (Full-time Hybrid, Morrisville, NC Based)

Alliance Health; Morrisville (NC); Hybrid; USD 108,000 - 137,000; Full time

Benefits offered by this job

Medical, Dental, Vision, Life

Long Term Disability

Retirement plan

Flexible schedules (hybrid/remote)

Paid time off

Dress flexibility

Job summary

Alliance Health seeks an AI Quality Engineer to ensure reliability, accuracy, and safety of AI-driven healthcare solutions throughout development. The role emphasizes designing evaluation frameworks for LLMs, agentic systems, and intelligent automation within clinical and operational settings.

This full-time hybrid position requires coming to the Morrisville home office on Wednesdays and residing within 50 miles of Morrisville, NC.

Qualifications

• Bachelor’s degree in Computer Science, Data Science, AI or related field with 3+ years in software/ML deployment.
• At least 2+ years of Python and ML libraries experience for developing and supporting AI/ML solutions.

Responsibilities

• Design and implement scalable AI Quality Engineering strategies for LLMs, agentic systems and enterprise AI platforms.
• Develop and maintain AI evaluation frameworks, datasets, scoring methodologies and automated testing workflows.
• Calibrate LLM judges to align with human expert preferences and clinical accuracy.
• Drive model and prompt optimization for specific use cases.
• Perform side-by-side performance analysis and trace agentic runs to identify failure modes.

Skills

AI Quality Engineering

Agentic & Generative AI Validation

AI Observability & Tracing

Advanced Programming

RAG & Vector DB Evaluation

LLMOps

AI-Augmented Productivity

Technical Communication

Education

Bachelor’s degree in Computer Science, Data Science, AI

Tools

LangSmith

GitHub Copilot

Job description

The AI Quality Engineer is responsible for ensuring the reliability, accuracy, and safety of AI-driven solutions from the earliest stages of development. This role is deeply integrated into the development lifecycle, focusing on building quality products by designing sophisticated evaluation frameworks for LLMs, agentic systems, and intelligent automation. The AI Quality Engineer ensures that AI models and AI Agents behave predictably, follow complex workflows, and meet the high standards required for clinical and operational healthcare environments.

This position is full-time hybrid. Selected candidate must come into the Home office in Morrisville on Wednesdays, for business meeting as needed, reside within 50 miles of Morrisville North Carolina.

Responsibilities & Duties

AI Evaluation & Framework Design

• Design and implement scalable AI Quality Engineering strategies supporting LLM solutions, agentic systems, and enterprise AI platforms
• Develop and maintain AI evaluation frameworks, including the creation of validation datasets, quality scoring methodologies, and automated testing workflows
• Execute advanced evaluation techniques, including running LLM-as-judge, code-based, and multi-turn evaluators on real production traces
• Calibrate LLM judges to ensure their assessments align with human expert preferences and clinical accuracy
• Drive model and prompt optimization by helping the team determine the most effective system prompts and optimal model selections for specific use cases

Agentic System Validation & Regression Testing

• Perform side-by-side performance analysis to quantify how changes to agent components impact overall performance, ensuring updates do not introduce regressions
• Trace full agentic runs and conversations to visualize every step an agent takes, utilizing distributed tracing and specialized AI assistants (e.g., Polly) to pinpoint failure modes
• Identify usage patterns and common failure modes using insights agents to provide executive-level summaries of system reliability

Collaborative Development & Human-in-the-Loop (HITL)

• Partner with AI Engineering, AIOps, and Security from the beginning of the development cycle to build quality into the architecture
• Enable Subject Matter Experts (SMEs) (Clinical, Operational, and Data teams) to review agent outputs and annotate traces, fostering a robust human-in-the-loop validation process
• Collaborate across agile delivery teams to coordinate testing activities, including sprint planning, defect management, and release readiness

Observability & Runtime Reliability

• Support runtime reliability by integrating quality controls with observability tools, monitoring for model drift, and assisting in incident response
• Leverage telemetry and distributed tracing to ensure AI-enabled systems maintain high availability and performance in production environments

Minimum Requirements

Education & Experience

Bachelor’s degree in Computer Science, Data Science, Artificial Intelligence, or a related field from an accredited college or university, and three (3) years of experience developing, implementing, and deploying software, machine learning, and/or artificial intelligence solutions. Experience must include integrating, deploying, monitoring, and maintaining AI/ML models and/or third-party AI/ML tools within production environments for internal business operations and/or customer-facing applications.

At least two (2) years of experience utilizing Python and related machine learning and artificial intelligence libraries and frameworks to develop, test, deploy, and support AI/ML solutions is required.

Knowledge, Skills, & Abilities

• AI Quality Engineering & Evaluation: designing and executing AI-native testing approaches, including LLM-as-judge, code-based evaluators, and multi-turn evaluators for agentic systems.
• Agentic & Generative AI Validation: evaluating the reasoning, tool-use, and workflow reliability of Agentic workflows and Large Language Models (LLMs) using both proprietary and open-source models.
• AI Observability & Tracing: using platforms like LangSmith for distributed tracing, monitoring agent runs, and analyzing production traces to identify failure modes.
• Advanced Programming & Frameworks: Python and libraries used for automated testing and evaluation.
• RAG & Vector Database Evaluation: validating retrieval-augmented generation (RAG) architectures.
• LLMOps & Lifecycle Management: managing prompt configurations, evaluation pipelines, and version control for models, prompts, and outputs to ensure consistency and prevent regressions.
• AI-Augmented Productivity: Ability to utilize AI tools (GitHub Copilot) to accelerate testing automation, documentation, and quality engineering tasks.
• Technical Communication & Leadership: Ability to translate complex AI performance metrics and evaluation findings into actionable insights for clinical, operational, and engineering stakeholders.

Employment for this position is contingent upon a satisfactory background check and credit check, which will be performed after acceptance of an offer of employment and prior to the employee's start date.

Salary Range

$107,607-$137,198/Annually

Exact compensation will be determined based on the candidate's education, experience, external market data and consideration of internal equity.

An excellent fringe benefit package accompanies the salary, which includes:

• Medical, Dental, Vision, Life, Long Term Disability
• Generous retirement savings plan
• Flexible work schedules including hybrid/remote options
• Paid time off including vacation, sick leave, holiday, management leave
• Dress flexibility

Education

Required

• Bachelors or better in Computer Science

Equal Opportunity Employer
This employer is required to notify all applicants of their rights pursuant to federal employment laws. For further information, please review the Know Your Rights notice from the Department of Labor.