Artificial Intelligence Quality Engineer (Full-time Hybrid, Morrisville, NC Based)

Alliance; United States; Hybrid; USD 108,000 - 137,000; Full time

Benefits offered by this job

Medical, Dental, Vision, Life, LTD

Generous retirement savings plan

Hybrid/Remote options

Paid time off: vacation, sick, holiday

Dress flexibility

Job summary

Alliance in Morrisville, NC is seeking an Artificial Intelligence Quality Engineer for a full-time hybrid role. You will design evaluation frameworks for LLMs, validate agentic systems, and ensure reliability in healthcare environments.

You will collaborate with AI Engineering, AIOps, and Security, mentor SMEs, and contribute to automated testing pipelines using Python and LangSmith.

Qualifications

• AI Quality Engineering & Evaluation: designing and executing AI-native testing approaches.
• Agentic & Generative AI Validation: evaluating agentic workflows and LLMs.
• AI Observability & Tracing: using tracing to identify failure modes.
• LLMOps & Lifecycle Management: managing prompts, configurations, and version control.
• Python for automated testing and evaluation.

Responsibilities

• Design and implement scalable AI Quality Engineering strategies for LLMs and enterprise AI platforms.
• Develop and maintain AI evaluation frameworks, datasets, and automated testing workflows.
• Perform side-by-side analyses to quantify changes and prevent regressions in agent components.
• Collaborate with AI Engineering, AIOps, and Security from early development to production.
• Enable SMEs to review agent outputs and annotate traces for validation.

Skills

AI Quality Engineering

Evaluation

Python

LangSmith

LLMOps

Observability

Distributed tracing

Technical communication

Leadership

Education

Bachelor's degree in Computer Science

Tools

LangSmith

GitHub Copilot

Job description

If you are a job seeker with a disability and require a reasonable accommodation to apply for one of our jobs, you will find the contact information to request the appropriate accommodation by visiting the following our Accessibility Accomodation for Applicants page.

Artificial Intelligence Quality Engineer (Full-time Hybrid, Morrisville, NC Based)

Locations

Showing 1 location

Home Office
Morrisville, NC 27560, USA

• Hybrid
• Full-time

Description

The AI Quality Engineer is responsible for ensuring the reliability, accuracy, and safety of AI-driven solutions from the earliest stages of development. This role is deeply integrated into the development lifecycle, focusing on building quality products by designing sophisticated evaluation frameworks for LLMs, agentic systems, and intelligent automation. The AI Quality Engineer ensures that AI models and AI Agents behave predictably, follow complex workflows, and meet the high standards required for clinical and operational healthcare environments.

This position is full-time hybrid. Selected candidate must come into the Home office in Morrisville on Wednesdays, for business meeting as needed, reside within 50 miles of Morrisville North Carolina.

AI Evaluation & Framework Design

• Design and implement scalable AI Quality Engineering strategies supporting LLM solutions, agentic systems, and enterprise AI platforms
• Develop and maintain AI evaluation frameworks, including the creation of validation datasets, quality scoring methodologies, and automated testing workflows
• Execute advanced evaluation techniques, including running LLM-as-judge, code-based, and multi-turn evaluators on real production traces
• Calibrate LLM judges to ensure their assessments align with human expert preferences and clinical accuracy
• Drive model and prompt optimization by helping the team determine the most effective system prompts and optimal model selections for specific use cases

Agentic System Validation & Regression Testing

• Perform side-by-side performance analysis to quantify how changes to agent components impact overall performance, ensuring updates do not introduce regressions
• Trace full agentic runs and conversations to visualize every step an agent takes, utilizing distributed tracing and specialized AI assistants (e.g., Polly) to pinpoint failure modes
• Identify usage patterns and common failure modes using insights agents to provide executive-level summaries of system reliability
• Partner with AI Engineering, AIOps, and Security from the beginning of the development cycle to build quality into the architecture
• Enable Subject Matter Experts (SMEs) (Clinical, Operational, and Data teams) to review agent outputs and annotate traces, fostering a robust human-in-the-loop validation process
• Collaborate across agile delivery teams to coordinate testing activities, including sprint planning, defect management, and release readiness

Observability & Runtime Reliability

• Support runtime reliability by integrating quality controls with observability tools, monitoring for model drift, and assisting in incident response
• Leverage telemetry and distributed tracing to ensure AI-enabled systems maintain high availability and performance in production environments

Education & Experience

Bachelor’s degree in Computer Science, Data Science, Artificial Intelligence, or a related field from an accredited college or university, and three (3) years of experience developing, implementing, and deploying software, machine learning, and/or artificial intelligence solutions. Experience must include integrating, deploying, monitoring, and maintaining AI/ML models and/or third-party AI/ML tools within production environments for internal business operations and/or customer-facing applications.

At least two (2) years of experience utilizing Python and related machine learning and artificial intelligence libraries and frameworks to develop, test, deploy, and support AI/ML solutions is required.

Knowledge, Skills, & Abilities

• AI Quality Engineering & Evaluation: designing and executing AI-native testing approaches, including LLM-as-judge, code-based evaluators, and multi-turn evaluators for agentic systems.
• Agentic & Generative AI Validation: evaluating the reasoning, tool-use, and workflow reliability of Agentic workflows and Large Language Models (LLMs) using both proprietary and open-source models.
• AI Observability & Tracing: using platforms like LangSmith for distributed tracing, monitoring agent runs, and analyzing production traces to identify failure modes.
• Advanced Programming & Frameworks: Python and libraries used for automated testing and evaluation.
• RAG & Vector Database Evaluation: validating retrieval-augmented generation (RAG) architectures.
• LLMOps & Lifecycle Management: managing prompt configurations, evaluation pipelines, and version control for models, prompts, and outputs to ensure consistency and prevent regressions.
• AI-Augmented Productivity: Ability to utilize AI tools (GitHub Copilot) to accelerate testing automation, documentation, and quality engineering tasks.
• Technical Communication & Leadership: Ability to translate complex AI performance metrics and evaluation findings into actionable insights for clinical, operational, and engineering stakeholders.

Employment for this position is contingent upon a satisfactory background check and credit check, which will be performed after acceptance of an offer of employment and prior to the employee's start date.

Salary Range

$107,607-$137,198/Annually

Exact compensation will be determined based on the candidate's education, experience, external market data and consideration of internal equity.

An excellent fringe benefit package accompanies the salary, which includes:

• Medical, Dental, Vision, Life, Long Term Disability
• Generous retirement savings plan
• Flexible work schedules including hybrid/remote options
• Paid time off including vacation, sick leave, holiday, management leave
• Dress flexibility

Qualifications

Education

Bachelors or better in Computer Science.

Equal Opportunity Employer This employer is required to notify all applicants of their rights pursuant to federal employment laws.For further information, please review the Know Your Rights notice from the Department of Labor.