Principal Evals Engineer

Client of Discovered MENA; Abu Dhabi; On-site; AED 480,000 - 720,000; Full time

Job summary

Client of Discovered MENA is seeking a Staff/Principal level Evals Engineer to define AI evaluation across the organisation. You will design shared evaluation architectures, build harnesses, datasets, and CI/CD integration, and stay hands-on building frameworks and running experiments.

You will focus on LLM evaluation, regression detection, RAG systems, and ensure decisions are data-driven with robust measurement and tooling across teams.

Qualifications

• Senior engineer with a track record building AI/testing infrastructure at scale.
• Experience owning evaluation frameworks rather than working inside a prebuilt platform.
• Strong Python engineering skills and hands-on gameplay with ML tooling.

Responsibilities

• Define AI evaluation strategy and reference architecture across the org.
• Build shared evaluation harnesses, libraries, and datasets with versioning.
• Develop automated evaluation and grading systems; detect regressions.
• Evaluate LLM output quality, grounding, and citation accuracy.
• Integrate evaluation into CI/CD pipelines and release processes.
• Lead adoption of rigorous evaluation standards across teams.

Skills

Python

LLM evaluation

RAG systems

Hallucination measurement

Regression detection

CI/CD

Reusable frameworks

Production AI systems

Rubric design

Statistics

LLM as a judge

Tools

LangChain

LangGraph

LangSmith

Langfuse

Ragas

DeepEval

Promptfoo

pgvector

Qdrant

Playwright

Cypress

Docker

Kubernetes

Azure

Job description

Discover the Role

As Principal Evals Engineer you will define how AI performance is measured across the organisation You will design and build the shared evaluation architecture used across multiple AI products and engineering teams This includes creating robust evaluation harnesses managing golden datasets automating grading detecting behavioural regressions and ensuring that engineering teams can make release decisions based on evidence rather than subjective judgement This is a senior individual contributor role You will remain highly hands on building frameworks running experiments and setting the technical standard for AI evaluation across the organisation

Discover the Responsibilities

• Define the overall AI evaluation strategy and reference architecture across the engineering organisation
• Build shared evaluation frameworks harnesses and reusable libraries
• Develop and maintain golden datasets and dataset versioning strategies
• Design automated evaluation and grading systems
• Implement regression detection for non deterministic AI behaviour
• Evaluate LLM output quality hallucination grounding faithfulness and citation accuracy
• Build evaluation frameworks for RAG and retrieval systems
• Evaluate agentic systems including tool usage multi step reasoning and failure recovery
• Design and calibrate LLM as a judge systems against human evaluation
• Establish production monitoring online evaluation and drift detection
• Develop adversarial testing methodologies covering prompt injection jailbreaks and data leakage
• Integrate AI evaluation into CI CD pipelines and release processes
• Help engineering teams understand and apply rigorous evaluation standards independently
• Contribute to building the broader evaluation engineering function as the organisation scales

Discover the Required

We are looking for a senior engineer with a strong track record of building AI or testing infrastructure at scale You should have experience owning evaluation frameworks rather than simply working inside an existing testing platform

Discover the Requirements

• Staff or Principal level engineering experience
• Strong hands on Python engineering skills
• Deep experience with LLM or Generative AI evaluation
• Experience evaluating RAG retrieval or agentic systems
• Strong understanding of hallucination grounding and output quality measurement
• Experience building regression detection for non deterministic AI systems
• Practical experience with LLM as a judge techniques
• Strong understanding of rubric design and calibration against human labels
• Statistical literacy including sampling confidence intervals significance and inter rater agreement
• Strong CI CD and software quality engineering experience
• Experience building reusable frameworks or shared engineering libraries
• Hands on experience working with production AI systems

Discover the Desired

Experience with one or more of the following would be highly valuable

• Ragas
• DeepEval
• Promptfoo
• Braintrust
• LangSmith
• Langfuse
• LangChain or LangGraph
• Vector databases such as pgvector or Qdrant
• AI red teaming or adversarial testing
• Voice or conversational AI evaluation
• Human evaluation programme design
• Data pipeline or performance testing
• Arabic language AI evaluation including golden datasets dialect coverage and right to left validation
• Experience working within regulated government or highly secure environments

Technology Environment

The primary language is Python with additional use of TypeScript and Java The wider environment includes custom evaluation harnesses LLM as a judge patterns golden datasets Langfuse Ragas DeepEval Promptfoo LangChain LangGraph Microsoft Agent Framework pgvector Qdrant Playwright Cypress Great Expectations Grafana Docker Kubernetes and Azure The team is pragmatic about tooling and is more interested in strong engineering judgement than experience with one specific framework