Generative AI QA Engineer | LLM Evaluation & E2E Testing

Expedite Talent Solutions; United States; On-site; USD 110,000 - 160,000; Full time

Job summary

Expedite Talent Solutions is seeking a QA/Automation Engineer to test Generative AI and conversational systems, perform LLM evaluation and benchmark testing, and validate RAG relevance and grounding. The role builds automation using Playwright/Selenium, conducts API and end-to-end testing, and strengthens AI safety and observability within CI/CD pipelines.

Ideal candidates will have HITL testing experience, strong SQL, Git and Python skills, and familiarity with CI/CD concepts in a collaborative

Qualifications

• QA & automation testing experience across web/app platforms.
• Experience with Playwright or Selenium for automated end-to-end tests.
• LLM evaluation, GenAI validation, and RAG testing.
• Prompt engineering and conversational AI testing.
• HITL testing, AI safety and observability practices.
• Familiarity with SQL, Git, Python and CI/CD concepts.

Responsibilities

• Test Generative AI and conversational systems.
• Perform LLM evaluation and benchmark testing.
• Validate RAG (retrieval quality, relevance, grounding).
• Build automation frameworks (Playwright/Selenium).
• Conduct API, regression, and E2E testing.
• Perform AI safety, red teaming, and bias validation.
• Support observability, monitoring, and HITL workflows.
• Collaborate with engineering, QA, and DevOps teams.
• Support CI/CD quality pipelines.

Skills

QA & automation testing

Prompt engineering

HITL testing

Observability

Python

CI/CD concepts

Tools

Playwright

Selenium

Postman

REST API testing

SQL

Git

GitHub Actions

Docker

Kubernetes

Job description

Expedite Talent Solutions is seeking a QA/Automation Engineer to test Generative AI and conversational systems, perform LLM evaluation and benchmark testing, and validate RAG relevance and grounding. The role builds automation using Playwright/Selenium, conducts API and end-to-end testing, and strengthens AI safety and observability within CI/CD pipelines.

Ideal candidates will have HITL testing experience, strong SQL, Git and Python skills, and familiarity with CI/CD concepts in a collaborative