AI QA Engineer – LLM, RAG & Conversational Testing

STATERA; Dubai; On-site; AED 290,000 - 469,000; Full time

Job summary

STATERA is seeking an AI Test Engineer in Dubai to automate testing for an AI-powered customer support and product troubleshooting solution. The role combines software test automation with hands-on evaluation of LLMs, RAG pipelines, and conversational agents.

You will build automated evaluations for RAG retrieval accuracy, context relevance, and response quality, while measuring faithfulness, relevance, and guardrail compliance.

Qualifications

• 4+ years in QA automation or SDET roles, including 1–2 years testing LLMs, chatbots, or agentic AI systems.
• Advanced proficiency in Python (pytest/ asyncio) and/or TypeScript or C#.
• Strong API testing and mocking experience with REST/GraphQL, WebSockets, Postman, WireMock, or custom mock servers.
• Experience evaluating RAG systems, semantic search, and vector database integrations.
• Ability to design reliable automated tests for multi-turn conversations and AI behavior.
• Experience with CI/CD and Azure DevOps.

Responsibilities

• Build automated evaluations for RAG retrieval accuracy, context relevance, semantic search precision, and response quality.
• Measure faithfulness, answer relevance, hallucination rates, and guardrail compliance.
• Test multi-turn troubleshooting flows across product issues such as connectivity failures, battery degradation, and performance faults.
• Simulate complex customer interactions to verify diagnostic state navigation and resolution paths.
• Prevent regressions when product documentation, firmware specifications, or telemetry models change.
• Automate API and end-to-end tests across the AI agent, customer support workflows, and warranty-related integrations.
• Run load and scalability tests on conversational APIs.
• Develop adversarial tests covering prompt injection, out-of-scope requests, and hallucination risks.
• Integrate AI evaluation and regression suites into Azure DevOps CI/CD pipelines.

Skills

Python / pytest / asyncio

TypeScript / C#

API testing & mocking

CI/CD & Azure DevOps

LLMs / AI testing

SDET / QA Automation

Tools

REST / GraphQL

Postman

WireMock

WebSockets

Azure DevOps CI/CD

Job description

STATERA is seeking an AI Test Engineer in Dubai to automate testing for an AI-powered customer support and product troubleshooting solution. The role combines software test automation with hands-on evaluation of LLMs, RAG pipelines, and conversational agents.

You will build automated evaluations for RAG retrieval accuracy, context relevance, and response quality, while measuring faithfulness, relevance, and guardrail compliance.