Senior AI QA Engineer

GCS; Abu Dhabi; Hybrid; AED 250,000 - 450,000; Full time

Job summary

GCS in the United Arab Emirates is hiring a Senior QA Automation Engineer to own quality for AI agentic systems, defining what good looks like when a system reasons, calls tools, and can be wrong in subtle ways. You’ll build end-to-end evaluation, and test non-deterministic AI across live production traffic.

You'll also cover API testing, security validations, and performance testing in a hybrid Abu Dhabi setup, collaborating with expert teams in a contract-based onboarding process.

Qualifications

• 5+ years in QA or quality engineering, with 2+ years testing ML, LLM, or AI agent systems.
• Hands-on with LLM/agent evaluation frameworks and LLM-as-judge.
• Agent observability and tracing experience.
• Expert Python with Pytest and API testing.

Responsibilities

• Test agentic systems end to end: tool-use and trajectory evaluation, grounding and citation verification, honest-failure and refusal calibration, guardrail and adversarial testing.
• Build evaluation for non-deterministic AI: scoring for hallucination, consistency, and accuracy against gold datasets.
• Own prompt and model regression suites that catch quality drift before it ships.
• Run continuous evaluation on live production traffic, with quality monitoring, alerting, and incident response.
• Cover fundamentals: integration and API testing, secure gateway validation (RBAC, PII), and performance/load testing.

Skills

Python

Pytest

Selenium

Playwright

API testing

LLM/AI evaluation

Quality engineering

Tools

DeepEval

TruLens

RAGAS

LangSmith

Langfuse

OpenTelemetry

Job description

About the role:

We're hiring a Senior QA Automation Engineer for one of the leading players in Sovereign AI, a UAE-based company with frontier-model access and agentic AI products live across critical sectors including large-scale construction, government, healthcare, and defense.

This is not traditional QA. You'll own quality for AI and agentic systems, defining what "good" looks like when a system reasons, calls tools, and can be wrong in subtle ways. If you've figured out how to test something whose output is different every time, this is for you.

What you'll do:

• Test agentic systems end to end: tool-use and trajectory evaluation, grounding and citation verification, honest-failure and refusal calibration, guardrail and adversarial testing (prompt injection, jailbreaks)
• Build evaluation for non-deterministic AI: scoring for hallucination, consistency, and accuracy against gold datasets, including LLM-as-judge calibrated against human judgement
• Own prompt and model regression suites that catch quality drift before it ships
• Run continuous evaluation on live production traffic, with quality monitoring, alerting, and incident response
• Cover the fundamentals too: integration and API testing, secure gateway validation (RBAC, PII), and performance/load testing

What we're looking for:

• 5+ years in QA or quality engineering, with 2+ years testing ML, LLM, or AI agent systems
• Hands-on with LLM/agent evaluation frameworks (DeepEval, TruLens, RAGAS, or custom Python evaluators) and LLM-as-judge
• Agent observability and tracing (LangSmith, Langfuse, OpenTelemetry)
• Expert Python (Pytest, custom harnesses), plus Selenium/Playwright and API testing

The setup:

• Abu Dhabi hybrid, contract-based
• Onboarding ASAP
• World-class environment: elite talent from FAANG and frontier AI labs