AI Quality Engineer

Expedite Talent Solutions; United States; On-site; USD 110,000 - 160,000; Full time

Job summary

Expedite Talent Solutions is seeking a QA/Automation Engineer to test Generative AI and conversational systems, perform LLM evaluation and benchmark testing, and validate RAG relevance and grounding. The role builds automation using Playwright/Selenium, conducts API and end-to-end testing, and strengthens AI safety and observability within CI/CD pipelines.

Ideal candidates will have HITL testing experience, strong SQL, Git and Python skills, and familiarity with CI/CD concepts in a collaborative

Qualifications

• QA & automation testing experience across web/app platforms.
• Experience with Playwright or Selenium for automated end-to-end tests.
• LLM evaluation, GenAI validation, and RAG testing.
• Prompt engineering and conversational AI testing.
• HITL testing, AI safety and observability practices.
• Familiarity with SQL, Git, Python and CI/CD concepts.

Responsibilities

• Test Generative AI and conversational systems.
• Perform LLM evaluation and benchmark testing.
• Validate RAG (retrieval quality, relevance, grounding).
• Build automation frameworks (Playwright/Selenium).
• Conduct API, regression, and E2E testing.
• Perform AI safety, red teaming, and bias validation.
• Support observability, monitoring, and HITL workflows.
• Collaborate with engineering, QA, and DevOps teams.
• Support CI/CD quality pipelines.

Skills

QA & automation testing

Prompt engineering

HITL testing

Observability

Python

CI/CD concepts

Tools

Playwright

Selenium

Postman

REST API testing

SQL

Git

GitHub Actions

Docker

Kubernetes

Job description

• Test Generative AI and conversational systems
• Perform LLM evaluation and benchmark testing
• Build automation frameworks (Playwright/Selenium)
• Conduct API, regression, and E2E testing
• Perform AI safety, red teaming, and bias validation
• Support observability, monitoring, and HITL workflows
• Collaborate with engineering, QA, and DevOps teams
• Support CI/CD quality pipelines

Key Responsibilities

• Test Generative AI and conversational systems
• Perform LLM evaluation and benchmark testing
• Validate RAG (retrieval quality, relevance, grounding)
• Build automation frameworks (Playwright/Selenium)
• Conduct API, regression, and E2E testing
• Perform AI safety, red teaming, and bias validation
• Support observability, monitoring, and HITL workflows
• Collaborate with engineering, QA, and DevOps teams
• Support CI/CD quality pipelines

Required Skills

• QA & automation testing
• Playwright/Selenium, API testing (Postman, REST)
• GenAI & RAG validation, LLM evaluation
• Prompt engineering, conversational AI testing
• HITL testing, AI safety & observability
• SQL, Git, Python, CI/CD concepts

Preferred

• OpenAI, Azure OpenAI, ChatGPT, Claude
• LangChain, CrewAI, LangGraph, MCP
• Vector DBs (Pinecone, ChromaDB, Weaviate)
• RAGAS, DeepEval

Python/JS, Docker, Kubernetes, GitHub Actions