Senior AI Safety & Evaluation Engineer (12-Month Onsite)

Discovered MENA; Abu Dhabi; Presencial; AED 320.000 - 520.000; Jornada completa

Descripción de la vacante

Discovered MENA is recruiting a Senior AI Evaluation & Safety Engineer for a leading UAE financial institution. You will design automated evaluation frameworks, assess LLMs, and implement safety guardrails for production AI applications within a regulated environment.

You will lead RAG/agentic AI evaluation efforts, build pipelines in Python, and collaborate with engineering teams to improve accuracy, reliability and security of Generative AI solutions onsite in Abu Dhabi.

Formación

• Bachelor's or Master's degree in CS/AI/ML/SE or related field.
• Strong professional experience in AI/ML engineering.
• Advanced Python programming skills for automated evaluation frameworks.
• Hands-on experience building or evaluating production Generative AI applications.
• Strong understanding of LLMs, RAG architectures, embeddings, vector databases and agentic AI systems.
• Experience with evaluation frameworks such as RAGAS, DeepEval, LangSmith, TruLens or equivalent.
• Experience with AI safety testing, adversarial testing, prompt injection or LLM red teaming.
• Understanding of LLM security vulnerabilities, guardrails and responsible AI principles.
• Experience integrating automated testing into engineering workflows.
• Experience in banking/financial services or regulated industries.
• Experience with Azure OpenAI / AWS Bedrock / Google Cloud AI services.
• Knowledge of LangChain / LangGraph / LlamaIndex.

Responsabilidades

• Design, develop and maintain automated evaluation frameworks for LLM-powered applications.
• Evaluate AI systems for accuracy, faithfulness, hallucinations, consistency and reliability.
• Develop golden datasets, evaluation benchmarks and automated testing pipelines.
• Implement LLM-as-a-Judge methodologies and automated scoring mechanisms.
• Define evaluation metrics, acceptance criteria and quality standards for production AI applications.
• Integrate evaluation and regression testing into AI development workflows.
• Develop evaluation frameworks for Retrieval-Augmented Generation (RAG) applications.
• Assess retrieval accuracy, context relevance, answer faithfulness and response quality.
• Evaluate AI agents, including tool calling and multi-step workflows.
• Identify failure scenarios and recommend improvements to AI performance.
• Collaborate with AI engineering teams to optimise system accuracy and reliability.
• Design and execute AI safety tests to identify vulnerabilities and unsafe model behaviour.
• Conduct adversarial testing, prompt injection detection and LLM red teaming.
• Evaluate risks relating to data exposure and confidential information leakage.
• Develop guardrails, safety controls and validation mechanisms.
• Support AI red teaming exercises and security assessments.
• Ensure AI applications align with responsible AI and enterprise security standards.
• Build scalable evaluation pipelines using Python and LLM evaluation frameworks.
• Work with tools such as RAGAS, DeepEval, LangSmith, TruLens or equivalent.
• Develop automated testing suites to evaluate AI applications across models/configurations.
• Monitor production AI applications for performance degradation, safety issues and unexpected behaviour.
• Document evaluation methodologies, test results, risks and remediation actions.
• Collaborate with cybersecurity, risk, compliance and governance teams.
• Contribute to development of AI evaluation and safety best practices.

Conocimientos

Python programming

AI/ML engineering

LLMs

RAG architectures

Embeddings

Agentic AI

Evaluation metrics

Automated testing

AI safety testing

Adversarial testing

LLM security

LangChain

LangGraph

LlamaIndex

Azure OpenAI

Educación

Bachelor's or Master's degree in Computer Science / AI / ML / Software Engineering

Herramientas

RAGAS

DeepEval

LangSmith

TruLens

LangChain

LlamaIndex

Descripción del empleo

Discovered MENA is recruiting a Senior AI Evaluation & Safety Engineer for a leading UAE financial institution. You will design automated evaluation frameworks, assess LLMs, and implement safety guardrails for production AI applications within a regulated environment.

You will lead RAG/agentic AI evaluation efforts, build pipelines in Python, and collaborate with engineering teams to improve accuracy, reliability and security of Generative AI solutions onsite in Abu Dhabi.