Senior AI Eval & Safety Engineer

Discovered MENA; Abu Dhabi; On-site; AED 320,000 - 520,000; Full time

Job summary

Discovered MENA is recruiting a Senior AI Evaluation & Safety Engineer for a leading UAE financial institution. You will design automated evaluation frameworks, assess LLMs, and implement safety guardrails for production AI applications within a regulated environment.

You will lead RAG/agentic AI evaluation efforts, build pipelines in Python, and collaborate with engineering teams to improve accuracy, reliability and security of Generative AI solutions onsite in Abu Dhabi.

Qualifications

• Bachelor's or Master's degree in CS/AI/ML/SE or related field.
• Strong professional experience in AI/ML engineering.
• Advanced Python programming skills for automated evaluation frameworks.
• Hands-on experience building or evaluating production Generative AI applications.
• Strong understanding of LLMs, RAG architectures, embeddings, vector databases and agentic AI systems.
• Experience with evaluation frameworks such as RAGAS, DeepEval, LangSmith, TruLens or equivalent.
• Experience with AI safety testing, adversarial testing, prompt injection or LLM red teaming.
• Understanding of LLM security vulnerabilities, guardrails and responsible AI principles.
• Experience integrating automated testing into engineering workflows.
• Experience in banking/financial services or regulated industries.
• Experience with Azure OpenAI / AWS Bedrock / Google Cloud AI services.
• Knowledge of LangChain / LangGraph / LlamaIndex.

Responsibilities

• Design, develop and maintain automated evaluation frameworks for LLM-powered applications.
• Evaluate AI systems for accuracy, faithfulness, hallucinations, consistency and reliability.
• Develop golden datasets, evaluation benchmarks and automated testing pipelines.
• Implement LLM-as-a-Judge methodologies and automated scoring mechanisms.
• Define evaluation metrics, acceptance criteria and quality standards for production AI applications.
• Integrate evaluation and regression testing into AI development workflows.
• Develop evaluation frameworks for Retrieval-Augmented Generation (RAG) applications.
• Assess retrieval accuracy, context relevance, answer faithfulness and response quality.
• Evaluate AI agents, including tool calling and multi-step workflows.
• Identify failure scenarios and recommend improvements to AI performance.
• Collaborate with AI engineering teams to optimise system accuracy and reliability.
• Design and execute AI safety tests to identify vulnerabilities and unsafe model behaviour.
• Conduct adversarial testing, prompt injection detection and LLM red teaming.
• Evaluate risks relating to data exposure and confidential information leakage.
• Develop guardrails, safety controls and validation mechanisms.
• Support AI red teaming exercises and security assessments.
• Ensure AI applications align with responsible AI and enterprise security standards.
• Build scalable evaluation pipelines using Python and LLM evaluation frameworks.
• Work with tools such as RAGAS, DeepEval, LangSmith, TruLens or equivalent.
• Develop automated testing suites to evaluate AI applications across models/configurations.
• Monitor production AI applications for performance degradation, safety issues and unexpected behaviour.
• Document evaluation methodologies, test results, risks and remediation actions.
• Collaborate with cybersecurity, risk, compliance and governance teams.
• Contribute to development of AI evaluation and safety best practices.

Skills

Python programming

AI/ML engineering

LLMs

RAG architectures

Embeddings

Agentic AI

Evaluation metrics

Automated testing

AI safety testing

Adversarial testing

LLM security

LangChain

LangGraph

LlamaIndex

Azure OpenAI

Education

Bachelor's or Master's degree in Computer Science / AI / ML / Software Engineering

Tools

RAGAS

DeepEval

LangSmith

TruLens

LangChain

LlamaIndex

Job description

We are partnering with a leading financial institution in the UAE that is investing heavily in Artificial Intelligence and Generative AI technologies.

We are looking for an experienced Senior AI Evaluation & Safety Engineer to join on a 12-month contract, supporting the development, evaluation and safe deployment of enterprise-scale AI applications.

Key Responsibilities

LLM Evaluation & Testing

• Design, develop and maintain automated evaluation frameworks for LLM-powered applications.
• Evaluate AI systems for accuracy, relevance, faithfulness, hallucinations, consistency and reliability.
• Develop golden datasets, evaluation benchmarks and automated testing pipelines.
• Implement LLM-as-a-Judge methodologies and automated scoring mechanisms.
• Define evaluation metrics, acceptance criteria and quality standards for production AI applications.
• Integrate evaluation and regression testing into AI development workflows.

RAG & Agentic AI Evaluation

• Develop evaluation frameworks for Retrieval-Augmented Generation (RAG) applications.
• Assess retrieval accuracy, context relevance, answer faithfulness and response quality.
• Evaluate AI agents, including tool calling, multi-step workflows, task completion and reliability.
• Identify failure scenarios and recommend improvements to AI application performance.
• Collaborate with AI engineering teams to optimise system accuracy and reliability.

AI Safety, Security & Red Teaming

• Design and execute AI safety tests to identify vulnerabilities and unsafe model behaviour.
• Conduct adversarial testing, including prompt injection, jailbreak attempts and manipulation of AI responses.
• Evaluate risks relating to sensitive data exposure, confidential information leakage and unauthorised actions.
• Develop and implement AI guardrails, safety controls and validation mechanisms.
• Support AI red teaming exercises and security assessments.
• Ensure AI applications align with responsible AI and enterprise security standards.

Evaluation Frameworks & Automation

• Build scalable evaluation pipelines using Python and relevant LLM evaluation frameworks.
• Work with tools such as RAGAS, DeepEval, LangSmith, TruLens, promptfoo or equivalent technologies.
• Develop automated testing suites to evaluate AI applications across different models and configurations.
• Monitor production AI applications for performance degradation, safety issues and unexpected behaviour.

AI Governance & Risk Management

• Support the implementation of responsible AI standards and evaluation controls.
• Ensure AI applications meet internal security, privacy and regulatory requirements.
• Document evaluation methodologies, test results, identified risks and remediation actions.
• Collaborate with cybersecurity, risk, compliance and governance teams.
• Contribute to the development of AI evaluation and safety best practices.

Required Skills & Experience

• Bachelor's or Master's degree in Computer Science, Artificial Intelligence, Machine Learning, Software Engineering or a related field.
• Strong professional experience in AI/ML engineering, applied AI or software engineering.
• Advanced Python programming skills, including experience developing automated evaluation and testing frameworks.
• Hands-on experience building or evaluating production Generative AI applications.
• Strong understanding of LLMs, RAG architectures, embeddings, vector databases and agentic AI systems.
• Practical experience implementing LLM evaluation metrics, including hallucination detection, faithfulness, relevance and response quality.
• Experience with evaluation frameworks such as RAGAS, DeepEval, LangSmith, TruLens or equivalent.
• Experience with AI safety testing, adversarial testing, prompt injection detection or LLM red teaming.
• Understanding of LLM security vulnerabilities, guardrails and responsible AI principles.
• Experience integrating automated testing and evaluation into engineering workflows.
• Strong analytical, debugging and problem-solving skills.
• Previous experience within banking, financial services, fintech or another highly regulated industry.
• Experience with Azure OpenAI, Azure AI Foundry, AWS Bedrock or Google Cloud AI services.
• Knowledge of LangChain, LangGraph, LlamaIndex or similar GenAI frameworks.
• Familiarity with NIST AI RMF and OWASP Top 10 for LLM Applications.
• Experience with LLMOps, model monitoring and AI observability.
• Understanding of data privacy, information security and regulatory compliance.
• Experience evaluating enterprise AI agents and multi-agent systems.

We are looking for a technically strong AI Engineer who has experience beyond simply building LLM applications and can demonstrate practical expertise in evaluating, testing and securing AI systems for production environments.

The ideal candidate will have personally developed automated evaluation frameworks, implemented AI safety testing and worked closely with engineering teams to improve the performance, reliability and security of Generative AI applications.

Experience delivering AI solutions within enterprise or regulated environments would be particularly valuable.

• Contract Duration: 12 months
• Working Arrangements: Onsite

This is an opportunity to work on advanced enterprise AI initiatives within a major financial institution, contributing to the development of secure, reliable and responsible Generative AI solutions.