QA Automation Engineer - GenAI / LLM Testing (Healthcare)

Luxoft

United States

Remote

USD 110,000 - 150,000

Full time

14 days+

Get a reply from this employer

Your resume and cover letter, tailored to exactly what they’re hiring for.

Get past ATS filters

Job summary

Luxoft is seeking an experienced AI Quality Engineer to design and implement quality strategies for AI-enabled applications, platforms, and services. The role focuses on automated testing, AI model evaluation, and ensuring the accuracy, reliability, safety, and compliance of GenAI, LLM, and ML solutions within enterprise environments.

You will collaborate with product, data science, and engineering teams to define acceptance criteria, produce quality metrics, and govern AI deployments across

Qualifications

  • 3–5+ years in Quality Engineering or Software Testing, incl. 3+ years in test automation and 2+ years testing AI/ML/GenAI/LLM solutions.
  • Experience designing automated test frameworks and validating AI-generated outputs.
  • Proficiency in Python or Java with knowledge of Agile, SDLC, CI/CD, and Responsible AI principles.
  • Strong analytical, communication, and cross-functional collaboration skills.
  • Experience testing RAG pipelines, chatbots, and LLM-based applications including prompt robustness and model drift.
  • Knowledge of AI evaluation frameworks, MLOps, AI governance, and cloud platforms (Azure, AWS, GCP).
  • Experience in healthcare or regulated environments is preferred.
  • Familiarity with accessibility testing, audit/compliance, and mentoring QA engineers is a plus.

Responsibilities

  • Design and execute testing strategies for AI/ML, GenAI, LLM, and automation solutions.
  • Develop and maintain automated testing frameworks covering functional, non‑functional, and AI-specific requirements.
  • Evaluate AI-generated outputs for accuracy, reliability, bias, safety, and Responsible AI compliance.
  • Collaborate with engineering, product, data science, and business teams to identify quality risks and acceptance criteria.
  • Produce quality metrics, testing documentation, governance evidence, and deployment/production monitoring recommendations.

Skills

Quality engineering
Test automation
AI/ML testing
GenAI/LLM testing
Python/Java
API testing
CI/CD
Agile/SDLC
Responsible AI

Tools

GitHub Copilot
Microsoft 365 Copilot

Job description

Project description Join Our Team: Innovating Health Care with Cutting-Edge Technology We are looking for an experienced AI Quality Engineer to design and implement quality engineering strategies for AI-enabled applications, platforms, and services. The role focuses on automated testing, AI model evaluation, and ensuring the accuracy, reliability, safety, and compliance of Generative AI, LLM, and machine learning solutions within enterprise environments.

Responsibilities
  • Design and execute comprehensive testing strategies for AI/ML, GenAI, LLM, and intelligent automation solutions.
  • Develop and maintain automated testing frameworks covering functional, non-functional, and AI-specific requirements.
  • Evaluate AI-generated outputs for accuracy, reliability, consistency, bias, fairness, safety, and Responsible AI compliance.
  • Collaborate with engineering, product, data science, and business teams to identify quality risks and establish acceptance criteria.
  • Produce quality metrics, testing documentation, governance evidence, and recommendations supporting AI deployment and production monitoring.
Skills
  • Must have Quality Engineering Test Automation AI / ML Testing Generative AI / LLM Testing Python / Java API Testing Integration Testing Data Validation Responsible AI AI Model Evaluation Test Framework Development CI/CD Agile / SDLC
Mandatory Skills Description
  • 3-5+ years of experience in Quality Engineering, Quality Assurance, or Software Testing, including 3+ years in test automation development and 2+ years testing AI, machine learning, Generative AI, LLM, or intelligent automation solutions.
  • Strong experience designing automated test frameworks, validating AI-generated outputs, and performing API, integration, and data testing.
  • Proficiency in Python, Java, or similar programming languages, with solid knowledge of Agile, SDLC, CI/CD, and Responsible AI principles, including fairness, bias detection, transparency, explainability, and human oversight.
  • Strong analytical, communication, and cross-functional collaboration skills are required.
  • Experience testing RAG pipelines, chatbots, conversational AI, AI agents, and LLM-based applications, including hallucination detection, prompt robustness, model drift, bias, and adversarial testing.
  • Knowledge of AI evaluation frameworks, test harnesses, MLOps, AI governance, and cloud platforms such as Azure, AWS, or GCP.
  • Experience within healthcare, payer, provider, pharmacy, claims, or other regulated environments is preferred.
  • Familiarity with accessibility testing, audit and compliance processes, and mentoring QA engineers is a plus.
AI Tool Expectations
  • Consistent use of enterprise-approved AI tools, including GitHub Copilot and Microsoft 365 Copilot, targeting a minimum of 90% weekly utilization to enhance testing, documentation, analysis, and delivery efficiency.
Nice to have
  • Exceptional communication skills.
  • Ability to deliver exceptional customer service with a positive attitude.
Other Languages

English: C1 Advanced Seniority Regular Remote United States, United States of America

Requirements

Req. VR-125940 Automated Testing Others Cross Industry Solutions 05/11/2026 Req. VR-125940

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Senior Java Test Automation Engineer with AI Expertise
Senior Java Test Automation Engineer with AI Expertise

EPAM Systems • United States

Remote
USD 120,000 - 170,000
AI Quality Architect
AI Quality Architect

EPAM Systems • United States

Remote
USD 140,000 - 210,000
Senior .NET Test Automation Engineer
Senior .NET Test Automation Engineer

EPAM Systems • United States

Remote
USD 120,000 - 160,000
Automation Test Engineer
Automation Test Engineer

Apt • Birmingham (AL)

On-site
USD 90,000 - 130,000
GMP Compliance & Equipment Qualification Specialist
GMP Compliance & Equipment Qualification Specialist

TechDigital Group • Summit (NJ)

On-site
USD 80,000 - 110,000
Quality Assurance Automation Engineer
Quality Assurance Automation Engineer

Career Soft Solutions Inc • Hartford (CT)

On-site
USD 90,000 - 120,000
QA Automation Engineer
QA Automation Engineer

GCS Recruitment • United States

Remote
USD 120,000 - 170,000
Quality Assurance Automation Engineer
Quality Assurance Automation Engineer

VetRamp • Dallas (TX)

Hybrid
USD 90,000 - 130,000
Employer-paid health insurance for you
Dental, vision, 401k and disability
Flexible remote/work-from-home options
+1
Lead QA Engineer for AI & LLM Workflows (Remote)
Lead QA Engineer for AI & LLM Workflows (Remote)

vibehackers • United States

Hybrid
USD 150,000 - 190,000
Generous annual bonus opportunity
401(k) with employer match
Medical Insurance
+2
Senior Product Software Engineer - AI Quality Engineering
Senior Product Software Engineer - AI Quality Engineering

C T Corporation System (DE) • Cape Girardeau (MO)

On-site
USD 140,000 - 190,000