QA AI Automation Engineer

Discovered MENA; Abu Dhabi; On-site; AED 100,000 - 167,000; Full time

Job summary

Discovered MENA in Abu Dhabi is seeking a QA AI Automation Engineer to own the testing strategy for internal AI models using real production and synthetic data. You will build Python-based automation test suites, validate AI outputs, and work with AI/ML and product teams to ensure reliability.

You will implement tests in CI/CD pipelines, track model performance, and define quality criteria per use case with robust dashboards for early regression detection.

Qualifications

• Bachelor's or master's degree in a related field.
• Strong Python expertise including scripting and testing frameworks like pytest.
• Proven experience testing AI/LLM outputs or validating ML systems.
• Experience handling data: cleaning real data, generating test data, golden datasets.
• Hands-on experience with CI/CD pipelines (GitLab CI or GitHub Actions).
• Solid understanding of SQL, NoSQL, data analysis, and vector search/embeddings; familiarity with FastAPI, Docker, Linux.

Responsibilities

• Build and maintain Python-based automation test suites from scratch for internal AI models.
• Validate AI model outputs for accuracy, consistency, edge cases, and failures.
• Collaborate with AI/ML and product teams to define quality criteria per use case.
• Establish performance metrics and dashboards; report regressions clearly.
• Integrate tests into CI/CD pipelines with automated gates to stop deployments when thresholds fail.
• Maintain model reliability over time and monitor performance through dashboards.

Skills

Python

pytest

APIs

CI/CD

SQL

NoSQL

FastAPI

Docker

Linux

LLM testing

RAGAS

LangSmith

Education

Bachelor's or Master's Degree

Tools

GitLab CI

GitHub Actions

LangSmith

RAGAS

DeepEval

Job description

We are hiring an experienced QA AI Automation Engineer to join a dynamic team in Abu Dhabi. This role owns the complete testing strategy by utilising a blend of real production data and synthetic test datasets to continuously evaluate AI models. It offers an exciting opportunity to work directly with modern AI technologies and internal LLM models in the UAE.

Discover the role

As a QA AI Automation Engineer, you will build and maintain Python based automation test suites from scratch. You will validate internal AI model output quality, accuracy, consistency, edge cases, and failure modes before and after every update. Collaborating with cross functional AI, ML, and product teams, you will establish performance metrics, integrate test suites into automated deployment pipelines, and maintain model reliability over time.

Discover the responsibilities

• Build and maintain Python based automation test suites and frameworks from scratch that continuously test internal AI models against real world and synthetic data
• Develop automation test suites for internal AI and LLM outputs covering accuracy, consistency, edge cases, and failure modes
• Build and manage test datasets combining real production data and generated synthetic scenarios to maximize test coverage
• Write automated test scripts including regression tests, golden dataset evaluations, and performance benchmarks for latency, cost, and throughput
• Integrate tests into CI CD pipelines with automated quality gates that stop deployments when thresholds drop
• Track and report model performance over time using clear dashboards and alerting systems to catch regressions early
• Work alongside AI, ML, and product teams to define quality criteria per use case and encode them into automated checks

Discover the requirements

• Bachelors or Masters Degree
• Strong Python expertise including scripting, testing frameworks such as pytest, and API integrations
• Proven experience testing AI and LLM outputs or validating ML systems using evaluation frameworks like RAGAS, DeepEval, LangSmith, or custom scoring
• Practical experience handling data, including cleaning real data, generating test data, and constructing golden datasets
• Hands on experience with CI CD pipelines including GitLab CI or GitHub Actions
• Solid understanding of SQL, NoSQL databases, data analysis, and vector search or embeddings Familiarity with FastAPI, Docker, Linux, Model Context Protocol, and GPU or CPU optimizations
• Understanding of model fine tuning concepts, prompt engineering, and observability tools Strong attention to detail, conceptual thinking, initiative, and a results oriented approach ISTQB CT AI certification is a plus