QA AI Automation Engineer: AI Model Validation & CI/CD

Discovered MENA; Abu Dhabi; On-site; AED 100,000 - 167,000; Full time

Job summary

Discovered MENA in Abu Dhabi is seeking a QA AI Automation Engineer to own the testing strategy for internal AI models using real production and synthetic data. You will build Python-based automation test suites, validate AI outputs, and work with AI/ML and product teams to ensure reliability.

You will implement tests in CI/CD pipelines, track model performance, and define quality criteria per use case with robust dashboards for early regression detection.

Qualifications

• Bachelor's or master's degree in a related field.
• Strong Python expertise including scripting and testing frameworks like pytest.
• Proven experience testing AI/LLM outputs or validating ML systems.
• Experience handling data: cleaning real data, generating test data, golden datasets.
• Hands-on experience with CI/CD pipelines (GitLab CI or GitHub Actions).
• Solid understanding of SQL, NoSQL, data analysis, and vector search/embeddings; familiarity with FastAPI, Docker, Linux.

Responsibilities

• Build and maintain Python-based automation test suites from scratch for internal AI models.
• Validate AI model outputs for accuracy, consistency, edge cases, and failures.
• Collaborate with AI/ML and product teams to define quality criteria per use case.
• Establish performance metrics and dashboards; report regressions clearly.
• Integrate tests into CI/CD pipelines with automated gates to stop deployments when thresholds fail.
• Maintain model reliability over time and monitor performance through dashboards.

Skills

Python

pytest

APIs

CI/CD

SQL

NoSQL

FastAPI

Docker

Linux

LLM testing

RAGAS

LangSmith

Education

Bachelor's or Master's Degree

Tools

GitLab CI

GitHub Actions

LangSmith

RAGAS

DeepEval

Job description

Discovered MENA in Abu Dhabi is seeking a QA AI Automation Engineer to own the testing strategy for internal AI models using real production and synthetic data. You will build Python-based automation test suites, validate AI outputs, and work with AI/ML and product teams to ensure reliability.

You will implement tests in CI/CD pipelines, track model performance, and define quality criteria per use case with robust dashboards for early regression detection.