Artificial Intelligence Quality Engineer (Full-time Hybrid, Morrisville, NC Based)

Alliance Health; Morrisville (NC); Hybride; USD 108 000 - 137 000; Plein temps

Avantages offerts par ce poste

Medical Insurance

Dental Insurance

Vision Insurance

Life Insurance

Long-Term Disability

Retirement Plan

Hybrid/Remote options

Paid Time Off

Dress Flexibility

Résumé du poste

Alliance Health in Morrisville, NC seeks an AI Quality Engineer to ensure reliability and safety of AI-driven healthcare solutions. The role integrates into development lifecycles, designing evaluation frameworks for LLMs, agentic systems, and automation to keep models aligned with clinical requirements.

This full‑time hybrid position requires in-office presence on Wednesdays and candidates living within 50 miles.

Qualifications

• Bachelor’s degree in Computer Science or related field with 3+ years in software/AI development.
• 3 years of experience developing, implementing, and deploying AI/ML solutions.
• 2+ years of Python experience for AI/ML development and testing.

Responsabilités

• Design and implement AI Quality Engineering strategies for LLMs and enterprise AI.
• Develop AI evaluation datasets, scoring methodologies, and automated testing workflows.
• Calibrate LLM judges to reflect clinical accuracy and human preferences.
• Collaborate with AI Engineering, AIOps, Security and SMEs in agile delivery.

Connaissances

AI Quality Engineering

Evaluation Frameworks

Python

LLMs

Agentic Systems

Observability

Distributed Tracing

QA Automation

RAG Evaluation

LLMOps

Formation

Bachelor's degree in Computer Science

Description du poste

If you are a job seeker with a disability and require a reasonable accommodation to apply for one of our jobs, you will find the contact information to request the appropriate accommodation by visiting the following our Accessibility Accomodation for Applicants page.

Artificial Intelligence Quality Engineer (Full-time Hybrid, Morrisville, NC Based)

Locations

Showing 1 location

Home Office
Morrisville, NC 27560, USA

• Hybrid
• Full-time

Description

The AI Quality Engineer is responsible for ensuring the reliability, accuracy, and safety of AI-driven solutions from the earliest stages of development. This role is deeply integrated into the development lifecycle, focusing on building quality products by designing sophisticated evaluation frameworks for LLMs, agentic systems, and intelligent automation. The AI Quality Engineer ensures that AI models and AI Agents behave predictably, follow complex workflows, and meet the high standards required for clinical and operational healthcare environments.

This position is full-time hybrid. Selected candidate must come into the Home office in Morrisville on Wednesdays, for business meeting as needed, reside within 50 miles of Morrisville North Carolina.

AI Evaluation & Framework Design

• Design and implement scalable AI Quality Engineering strategies supporting LLM solutions, agentic systems, and enterprise AI platforms
• Develop and maintain AI evaluation frameworks, including the creation of validation datasets, quality scoring methodologies, and automated testing workflows
• Execute advanced evaluation techniques, including running LLM-as-judge, code-based, and multi-turn evaluators on real production traces
• Calibrate LLM judges to ensure their assessments align with human expert preferences and clinical accuracy
• Drive model and prompt optimization by helping the team determine the most effective system prompts and optimal model selections for specific use cases

Agentic System Validation & Regression Testing

• Perform side-by-side performance analysis to quantify how changes to agent components impact overall performance, ensuring updates do not introduce regressions
• Trace full agentic runs and conversations to visualize every step an agent takes, utilizing distributed tracing and specialized AI assistants (e.g., Polly) to pinpoint failure modes
• Identify usage patterns and common failure modes using insights agents to provide executive-level summaries of system reliability
• Partner with AI Engineering, AIOps, and Security from the beginning of the development cycle to build quality into the architecture
• Enable Subject Matter Experts (SMEs) (Clinical, Operational, and Data teams) to review agent outputs and annotate traces, fostering a robust human-in-the-loop validation process
• Collaborate across agile delivery teams to coordinate testing activities, including sprint planning, defect management, and release readiness

Observability & Runtime Reliability

• Support runtime reliability by integrating quality controls with observability tools, monitoring for model drift, and assisting in incident response
• Leverage telemetry and distributed tracing to ensure AI-enabled systems maintain high availability and performance in production environments

Education & Experience

Bachelor’s degree in Computer Science, Data Science, Artificial Intelligence, or a related field from an accredited college or university, and three (3) years of experience developing, implementing, and deploying software, machine learning, and/or artificial intelligence solutions. Experience must include integrating, deploying, monitoring, and maintaining AI/ML models and/or third-party AI/ML tools within production environments for internal business operations and/or customer-facing applications.

At least two (2) years of experience utilizing Python and related machine learning and artificial intelligence libraries and frameworks to develop, test, deploy, and support AI/ML solutions is required.

Knowledge, Skills, & Abilities

• AI Quality Engineering & Evaluation: designing and executing AI-native testing approaches, including LLM-as-judge, code-based evaluators, and multi-turn evaluators for agentic systems.
• Agentic & Generative AI Validation: evaluating the reasoning, tool-use, and workflow reliability of Agentic workflows and Large Language Models (LLMs) using both proprietary and open-source models.
• AI Observability & Tracing: using platforms like LangSmith for distributed tracing, monitoring agent runs, and analyzing production traces to identify failure modes.
• Advanced Programming & Frameworks: Python and libraries used for automated testing and evaluation.
• RAG & Vector Database Evaluation: validating retrieval-augmented generation (RAG) architectures.
• LLMOps & Lifecycle Management: managing prompt configurations, evaluation pipelines, and version control for models, prompts, and outputs to ensure consistency and prevent regressions.
• AI-Augmented Productivity: Ability to utilize AI tools (GitHub Copilot) to accelerate testing automation, documentation, and quality engineering tasks.
• Technical Communication & Leadership: Ability to translate complex AI performance metrics and evaluation findings into actionable insights for clinical, operational, and engineering stakeholders.

Employment for this position is contingent upon a satisfactory background check and credit check, which will be performed after acceptance of an offer of employment and prior to the employee's start date.

Salary Range

$107,607-$137,198/Annually

Exact compensation will be determined based on the candidate's education, experience, external market data and consideration of internal equity.

An excellent fringe benefit package accompanies the salary, which includes:

• Medical, Dental, Vision, Life, Long Term Disability
• Generous retirement savings plan
• Flexible work schedules including hybrid/remote options
• Paid time off including vacation, sick leave, holiday, management leave
• Dress flexibility

Qualifications

Education

Bachelors or better in Computer Science.

Equal Opportunity Employer This employer is required to notify all applicants of their rights pursuant to federal employment laws.For further information, please review the Know Your Rights notice from the Department of Labor.