Senior AI Evaluation Engineer – LLMs & Agentic AI (Qatar)

Management Solutions International (MSI)

Doha

On-site

QAR 350,000 - 500,000

Full time

29 hours ago
Be an early applicant

A complete application in a minute

Tailored resume and cover letter, ready to send.

Get past ATS filters

Job summary

Management Solutions International (MSI) seeks a Senior AI Evaluation Engineer to lead evaluation, testing and validation of advanced AI and LLM systems in production contexts. This role emphasizes ensuring accuracy, reliability and safety of AI-powered applications.

You will design evaluation frameworks, benchmarks and automated pipelines, and guide continuous improvement through statistical analysis and controlled experiments. Doha, Qatar-based with senior responsibilities.

Qualifications

  • 7+ years of experience in AI evaluation, testing and validation.
  • Master's degree or higher in Computer Science, Software Engineering or related field.
  • Strong expertise in evaluating AI systems for factual accuracy, reliability and production readiness.

Responsibilities

  • Design and implement comprehensive evaluation frameworks for LLMs, Generative AI and RAG-based applications.
  • Develop datasets, test cases, ground-truth benchmarks and scoring rubrics.
  • Build automated evaluation pipelines and statistical validation methodologies.
  • Lead benchmarking and experimental validation to identify performance gaps and improvement opportunities.

Skills

LLM evaluation
Statistical analysis
Benchmarking
Experimental design
AB testing
Evaluation frameworks
Automation testing

Education

Master's Degree in CS/SE

Tools

RAGAS
DeepEval
TruLens
OpenAI Evals
LangSmith
Arize Phoenix

Job description

Management Solutions International (MSI) seeks a Senior AI Evaluation Engineer to lead evaluation, testing and validation of advanced AI and LLM systems in production contexts. This role emphasizes ensuring accuracy, reliability and safety of AI-powered applications.

You will design evaluation frameworks, benchmarks and automated pipelines, and guide continuous improvement through statistical analysis and controlled experiments. Doha, Qatar-based with senior responsibilities.

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Artificial Intelligence Engineer
Artificial Intelligence Engineer

Management Solutions International (MSI) • Doha

On-site
QAR 350,000 - 500,000
Senior AI Evals Engineer — Quality & Evaluation Lead
Senior AI Evals Engineer — Quality & Evaluation Lead

Alpha10x • Doha

On-site
QAR 360,000 - 600,000
Doha On-Site Senior LLM Engineer (Production AI)
Doha On-Site Senior LLM Engineer (Production AI)

WorkWavez • Qatar

On-site
QAR 450,000 - 600,000
Senior AI Developer – Doha, Qatar
Senior AI Developer – Doha, Qatar

WorkWavez • Qatar

On-site
QAR 450,000 - 600,000
Senior Engineer - Artificial Intelligence (AI)
Senior Engineer - Artificial Intelligence (AI)

Talent Leaders Inc. • Doha

On-site
QAR 437,000 - 692,000
Senior AI Architect: Production ML & RAG Systems Lead
Senior AI Architect: Production ML & RAG Systems Lead

Cplusoft • Doha

On-site
QAR 400,000 - 800,000
AI QA Architect: End-to-End LLM Testing & Automation
AI QA Architect: End-to-End LLM Testing & Automation

Arabic.AI • Doha

On-site
QAR 180,000 - 240,000
AI/ML Engineer: Deploy & Optimize Models
AI/ML Engineer: Deploy & Optimize Models

Maseera • Doha

On-site
QAR 180,000 - 360,000
Senior AI Engineer - Production-Ready Agentic Systems Doha
Senior AI Engineer - Production-Ready Agentic Systems Doha

Alpha10x • Doha

On-site
QAR 300,000 - 540,000
AI Software Engineer - Full Stack Developer (Java, React)
AI Software Engineer - Full Stack Developer (Java, React)

EPAM Systems • Qatar

On-site
QAR 240,000 - 360,000
Top locations