AI Quality Engineer: Healthcare AI Validation & Testing

CitiusTech; Chicago (IL); On-site; USD 140,000 - 195,000; Full time

Job summary

CitiusTech is seeking a hands-on leader to drive AI Quality Engineering initiatives across enterprise AI platforms and LLM-enabled solutions. You will design scalable testing frameworks, AI-native validation, runtime observability, and cross-functional collaboration with AI, security, governance, and product teams.

The role emphasizes mentorship of Quality Engineers, modernization of testing practices, and responsible AI deployment within regulated healthcare environments.

Qualifications

• Five or more years of experience in Quality Engineering, Quality Assurance, software testing, enterprise application delivery, technology operations, or related technology functions required.
• Two or more years of experience providing technical leadership for testing initiatives, automation programs, or enterprise technology delivery projects required.
• Experience supporting Quality Engineering or Quality Assurance across enterprise platforms, APIs, healthcare applications, operational workflows, or integrated business systems required.
• Strong knowledge of software development life cycle (SDLC), Agile methodologies, test automation frameworks, defect management, release validation, and production support processes required.
• Experience validating AI-powered applications, intelligent automation, machine learning, large language model (LLM), or AI-enabled business workflows preferred.
• Experience with AI Quality Engineering practices, AI-assisted testing, runtime observability, monitoring, telemetry, or reliability engineering preferred.
• Strong analytical, problem‑solving, organizational, communication, collaboration, and leadership skills required.
• Demonstrated ability to manage multiple priorities and deliver results within fast‑paced, highly collaborative enterprise environments required.
• Experience in healthcare technology, payer operations, clinical workflows, or other regulated industries supporting AI governance and responsible AI deployment preferred.

Responsibilities

• Design, develop, and execute AI Quality Engineering strategies supporting AI-powered applications, large language model (LLM) solutions, intelligent automation, agentic systems, and enterprise AI platforms.
• Build and implement scalable AI Quality Engineering practices, including AI-native testing approaches, validation processes, runtime quality controls, reusable testing accelerators, and automated testing frameworks.
• Lead AI validation activities including functional testing, prompt validation, workflow testing, regression testing, release validation, runtime quality assurance, and production reliability support.
• Partner with AI Engineering, AIOps, LLMOps, Security, Governance, Clinical, Data, and Product teams to deliver scalable AI Quality Engineering processes across enterprise AI initiatives.
• Support runtime reliability through observability, telemetry, distributed tracing, monitoring, drift detection, incident response, and operational quality assurance for AI-enabled systems.
• Develop and maintain AI evaluation frameworks, validation datasets, quality scoring methodologies, and automated testing workflows that improve the reliability and scalability of AI solutions.
• Collaborate with Clinical, Operational, and Engineering stakeholders to validate healthcare workflows, payer operations, and AI-enabled business processes while supporting responsible AI deployment through human‑in‑the‑loop validation practices.
• Coordinate testing activities across Agile delivery teams, including sprint planning, test execution, defect management, issue tracking, release readiness, risk identification, and production support.
• Mentor Quality Engineers and provide technical guidance that promotes engineering excellence, AI-enabled testing modernization, continuous improvement, and adoption of modern Quality Engineering practices.
• Research, evaluate, and recommend emerging AI Quality Engineering, testing automation, observability, and runtime assurance technologies to continuously improve enterprise AI quality capabilities.

Job description

CitiusTech is seeking a hands-on leader to drive AI Quality Engineering initiatives across enterprise AI platforms and LLM-enabled solutions. You will design scalable testing frameworks, AI-native validation, runtime observability, and cross-functional collaboration with AI, security, governance, and product teams.

The role emphasizes mentorship of Quality Engineers, modernization of testing practices, and responsible AI deployment within regulated healthcare environments.