Artificial Intelligence Engineer

Management Solutions International (MSI)

Doha

On-site

QAR 350,000 - 500,000

Full time

29 hours ago
Be an early applicant

Get a reply from this employer

Your resume and cover letter, tailored to exactly what they’re hiring for.

Get past ATS filters

Job summary

Management Solutions International (MSI) seeks a Senior AI Evaluation Engineer to lead evaluation, testing and validation of advanced AI and LLM systems in production contexts. This role emphasizes ensuring accuracy, reliability and safety of AI-powered applications.

You will design evaluation frameworks, benchmarks and automated pipelines, and guide continuous improvement through statistical analysis and controlled experiments. Doha, Qatar-based with senior responsibilities.

Qualifications

  • 7+ years of experience in AI evaluation, testing and validation.
  • Master's degree or higher in Computer Science, Software Engineering or related field.
  • Strong expertise in evaluating AI systems for factual accuracy, reliability and production readiness.

Responsibilities

  • Design and implement comprehensive evaluation frameworks for LLMs, Generative AI and RAG-based applications.
  • Develop datasets, test cases, ground-truth benchmarks and scoring rubrics.
  • Build automated evaluation pipelines and statistical validation methodologies.
  • Lead benchmarking and experimental validation to identify performance gaps and improvement opportunities.

Skills

LLM evaluation
Statistical analysis
Benchmarking
Experimental design
AB testing
Evaluation frameworks
Automation testing

Education

Master's Degree in CS/SE

Tools

RAGAS
DeepEval
TruLens
OpenAI Evals
LangSmith
Arize Phoenix

Job description

Senior AI Evaluation Engineer – LLMs & Agentic AI


Location: Qatar


Experience: 7+ Years


Education: Master's Degree or Higher in Computer Science, Software Engineering or a related Engineering discipline.


About the Role


We are looking for an experienced Senior AI Evaluation Engineer to lead the evaluation, testing, validation and continuous improvement of advanced AI and Large Language Model (LLM) systems.


This is not a conventional Machine Learning Engineer or Generative AI Developer position focused primarily on building and deploying AI applications.


The successful candidate will specialize in determining whether AI systems are actually performing correctly, reliably and consistently in real-world production environments.


You will design and implement evaluation frameworks, benchmark datasets, automated testing pipelines and statistical validation methodologies to measure the quality, accuracy, factual reliability and performance of LLM-powered applications and agentic AI systems.


Key Responsibilities


1. LLM & Generative AI Evaluation



  • Design and implement comprehensive evaluation frameworks for Large Language Models (LLMs), Generative AI and RAG-based applications.

  • Develop evaluation methodologies to measure factual accuracy, relevance, hallucinations, contextual understanding, consistency and response quality.

  • Implement and validate LLM-as-a-Judge approaches for automated evaluation and scoring.

  • Evaluate different LLM models, prompts and configurations through structured benchmarking and comparative testing.


2. Evaluation Frameworks & Benchmarking



  • Build and maintain automated evaluation pipelines using frameworks such as RAGAS, DeepEval, TruLens, OpenAI Evals, LangSmith, Arize Phoenix or equivalent tools.

  • Design evaluation datasets, test cases, ground-truth benchmarks and scoring rubrics.

  • Develop repeatable testing processes for assessing model performance across different use cases.

  • Establish evaluation metrics and quality thresholds to determine whether AI systems are ready for production deployment.


3. Statistical Analysis & Experimental Validation



  • Apply statistical methods to assess model performance and reliability.

  • Design and conduct controlled experiments, A/B testing and comparative model evaluations.

  • Perform hypothesis testing, statistical significance analysis and confidence interval calculations.

  • Analyse evaluation results to identify performance gaps, model weaknesses and opportunities for improvement.

  • Apply human

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Senior AI Evaluation Engineer – LLMs & Agentic AI (Qatar)
Senior AI Evaluation Engineer – LLMs & Agentic AI (Qatar)

Management Solutions International (MSI) • Doha

On-site
QAR 350,000 - 500,000
Senior Engineer - Artificial Intelligence (AI)
Senior Engineer - Artificial Intelligence (AI)

Talent Leaders Inc. • Doha

On-site
QAR 437,000 - 692,000
Senior AI Evals Engineer
Senior AI Evals Engineer

Alpha10x • Doha

On-site
QAR 360,000 - 600,000
Senior AI Developer – Doha, Qatar
Senior AI Developer – Doha, Qatar

WorkWavez • Qatar

On-site
QAR 450,000 - 600,000
Senior Engineer – AI
Senior Engineer – AI

MEEZA QSTP • Doha

On-site
QAR 400,000 - 750,000
Senior AI Engineer
Senior AI Engineer

Cplusoft • Doha

On-site
QAR 400,000 - 800,000
Senior AI Engineer
Senior AI Engineer

Alpha10x • Doha

On-site
QAR 300,000 - 540,000
Senior QA Engineer - AI Products (6 Months Contract)
Senior QA Engineer - AI Products (6 Months Contract)

Arabic.AI • Doha

On-site
QAR 180,000 - 240,000
Senior AI Engineer: Enterprise LLMs & RAG Architect
Senior AI Engineer: Enterprise LLMs & RAG Architect

MEEZA QSTP • Doha

On-site
QAR 400,000 - 750,000
AI Software Engineer - Full Stack Developer (Java, React)
AI Software Engineer - Full Stack Developer (Java, React)

EPAM Systems • Qatar

On-site
QAR 240,000 - 360,000
Top locations