Principal AI Evaluation Architect

Client of Discovered MENA; Abu Dhabi; On-site; AED 480,000 - 720,000; Full time

Job summary

Client of Discovered MENA is seeking a Staff/Principal level Evals Engineer to define AI evaluation across the organisation. You will design shared evaluation architectures, build harnesses, datasets, and CI/CD integration, and stay hands-on building frameworks and running experiments.

You will focus on LLM evaluation, regression detection, RAG systems, and ensure decisions are data-driven with robust measurement and tooling across teams.

Qualifications

• Senior engineer with a track record building AI/testing infrastructure at scale.
• Experience owning evaluation frameworks rather than working inside a prebuilt platform.
• Strong Python engineering skills and hands-on gameplay with ML tooling.

Responsibilities

• Define AI evaluation strategy and reference architecture across the org.
• Build shared evaluation harnesses, libraries, and datasets with versioning.
• Develop automated evaluation and grading systems; detect regressions.
• Evaluate LLM output quality, grounding, and citation accuracy.
• Integrate evaluation into CI/CD pipelines and release processes.
• Lead adoption of rigorous evaluation standards across teams.

Skills

Python

LLM evaluation

RAG systems

Hallucination measurement

Regression detection

CI/CD

Reusable frameworks

Production AI systems

Rubric design

Statistics

LLM as a judge

Tools

LangChain

LangGraph

LangSmith

Langfuse

Ragas

DeepEval

Promptfoo

pgvector

Qdrant

Playwright

Cypress

Docker

Kubernetes

Azure

Job description

Client of Discovered MENA is seeking a Staff/Principal level Evals Engineer to define AI evaluation across the organisation. You will design shared evaluation architectures, build harnesses, datasets, and CI/CD integration, and stay hands-on building frameworks and running experiments.

You will focus on LLM evaluation, regression detection, RAG systems, and ensure decisions are data-driven with robust measurement and tooling across teams.