AI Quality Engineer — Hybrid, Impactful AI Validation

Alliance Health; Morrisville (NC); Hybrid; USD 108.000 - 137.000; Vollzeit

Benefits dieser Stelle

Medical, Dental, Vision, Life

Long Term Disability

Retirement plan

Flexible schedules (hybrid/remote)

Paid time off

Dress flexibility

Zusammenfassung

Alliance Health seeks an AI Quality Engineer to ensure reliability, accuracy, and safety of AI-driven healthcare solutions throughout development. The role emphasizes designing evaluation frameworks for LLMs, agentic systems, and intelligent automation within clinical and operational settings.

This full-time hybrid position requires coming to the Morrisville home office on Wednesdays and residing within 50 miles of Morrisville, NC.

Qualifikationen

• Bachelor’s degree in Computer Science, Data Science, AI or related field with 3+ years in software/ML deployment.
• At least 2+ years of Python and ML libraries experience for developing and supporting AI/ML solutions.

Aufgaben

• Design and implement scalable AI Quality Engineering strategies for LLMs, agentic systems and enterprise AI platforms.
• Develop and maintain AI evaluation frameworks, datasets, scoring methodologies and automated testing workflows.
• Calibrate LLM judges to align with human expert preferences and clinical accuracy.
• Drive model and prompt optimization for specific use cases.
• Perform side-by-side performance analysis and trace agentic runs to identify failure modes.

Kenntnisse

AI Quality Engineering

Agentic & Generative AI Validation

AI Observability & Tracing

Advanced Programming

RAG & Vector DB Evaluation

LLMOps

AI-Augmented Productivity

Technical Communication

Ausbildung

Bachelor’s degree in Computer Science, Data Science, AI

Tools

LangSmith

GitHub Copilot

Jobbeschreibung

Alliance Health seeks an AI Quality Engineer to ensure reliability, accuracy, and safety of AI-driven healthcare solutions throughout development. The role emphasizes designing evaluation frameworks for LLMs, agentic systems, and intelligent automation within clinical and operational settings.

This full-time hybrid position requires coming to the Morrisville home office on Wednesdays and residing within 50 miles of Morrisville, NC.