Sr AI Quality & Reliability Engineer

Talentola; Chicago (IL); Hybrid; USD 150,000 - 190,000; Full time

Job summary

Talentola is seeking a Sr AI Quality & Reliability Engineer to lead hands-on AI quality initiatives across enterprise platforms. The role focuses on AI validation, testing, observability, and runtime assurance to enable responsible deployment of AI-powered solutions.

The position is Hybrid in Chicago and involves collaboration with AI, Clinical, Operations, and Engineering teams to accelerate AI adoption while ensuring quality and governance.

Qualifications

• Design and implement AI quality engineering practices and validation approaches.
• Coordinate AI validation activities, defect management, and release support.
• Mentor quality engineers and analysts across delivery teams.

Responsibilities

• Design and execute AI Quality Engineering activities for AI-powered applications.
• Support AI validation, testing, and runtime quality assurance across enterprise platforms.
• Collaborate with Clinical, Operational, and Engineering teams to ensure governance-aligned testing.

Skills

AI Quality Engineering

AI validation

Test automation

Observability

Cross-functional collaboration

Education

Engineering Degree BE/ME/BTech/MTech/BSc/MSc.

Tools

Telemetry tools

Distributed tracing

LLMOps

Job description

What is in it for you?

The Sr AI Quality & Reliability Engineer will drive the hands-on design, development, and execution of AI Quality Engineering initiatives supporting enterprise AI transformation efforts.
The role will design and implement AI Quality Engineering practices, AI validation processes, AI-assisted testing approaches, runtime quality controls, and scalable testing frameworks supporting responsible deployment of AI-powered business solutions.
This role combines hands-on quality engineering, AI-enabled testing modernization, healthcare workflow validation, technical mentorship, and cross-functional collaboration. This role will help advance Quality Engineering capabilities beyond traditional software testing practices toward AI-native validation, AI-assisted testing, runtime observability, reliability engineering, and modern AI quality engineering practices.

Job Description

AI Quality Engineering & Technical Execution
Design and execute AI Quality Engineering activities supporting AI-powered applications, LLM-enabled workflows, intelligent automation solutions, agentic systems, and enterprise AI platforms.
Build and implement AI Quality Engineering practices, including AI-native testing approaches, validation processes, runtime quality controls, reusable testing accelerators, and scalable testing workflows.
Drive modernization of traditional Quality Engineering practices to address AI-enabled workflows, intelligent orchestration, and evolving healthcare operational workflows.
Support implementation of AI quality standards, validation approaches, release readiness processes, and governance-aligned testing practices supporting enterprise AI adoption.
Coordinate AI validation activities, defect management, test execution, release support, and quality improvement initiatives across AI-enabled systems. AI Validation, Testing & Runtime Assurance
Support AI validation activities, including functional validation, prompt testing, workflow testing, regression testing, runtime quality assurance, and production reliability support.
Partner with AI Engineering, AIOps, LLMOps, Security, Governance, Clinical, and Data teams to support scalable AI Quality Engineering processes across enterprise AI delivery initiatives.
Support runtime quality and reliability practices, including telemetry alignment, distributed tracing, observability, monitoring coordination, incident support, release validation, and runtime reliability improvement efforts across AI-enabled systems.
Support AI evaluation frameworks, validation datasets, quality scoring approaches, and automated testing workflows supporting scalable AI Quality Engineering practices.
Support drift detection, runtime monitoring, and operational quality assurance initiatives across AI-enabled systems.
Support human-in-the-loop validation processes and operational review workflows supporting responsible AI deployment.
Drive adoption of AI-assisted testing approaches, intelligent automation, reusable testing accelerators, and scalable test automation practices.
Support observability initiatives improving reliability, traceability, and confidence across AI-enabled systems.
Collaborate with Clinical, Operational, and Engineering stakeholders to support validation of healthcare workflows, payer operations, and AI-enabled business processes. Delivery Coordination & Operational Support
Contribute to delivery coordination across AI Quality Engineering initiatives, including sprint execution, testing coordination, issue tracking, risk identification, and release support activities.
Partner with stakeholders to evaluate testing readiness, implementation dependencies, quality risks, and operational support considerations for AI initiatives.
Support tooling evaluations, automation initiatives, and modernization efforts supporting AI Quality Engineering maturity.
Help establish scalable testing processes, reusable quality engineering assets, and operational support models across AI delivery teams.
Support adoption of modern AI Quality Engineering practices across engineering and delivery organizations. Collaboration, Mentorship & Continuous Improvement
Mentor quality engineers and analysts and provide technical guidance to delivery teams while fostering a collaborative, continuously learning, and engineering-focused culture.
Communicate testing risks, implementation issues, delivery tradeoffs, and operational recommendations to technical and business stakeholders.
Promote engineering discipline, continuous improvement, responsible AI adoption, and operational accountability across AI Quality Engineering initiatives.
Research and evaluate emerging AI quality engineering, testing, observability, automation, and runtime assurance technologies supporting continuous improvement

Work Mode

Hybrid -3 days onsite at either of the location: Chicago/Irving

Educational Qualifications

• Engineering Degree BE/ME/BTech/MTech/BSc/MSc.
• Technical certification in multiple technologies is desirable.