AI Quality & Reliability Engineer – QA/SRE

D4 Insight; Abu Dhabi; Hybrid; AED 420,000 - 650,000; Full time

Job summary

D4 Insight is seeking an AI Quality & Reliability Engineer in Abu Dhabi to own production readiness for cloud AI solutions. This hybrid role blends automated testing, AI evaluation, and SRE practices to ensure resilient, secure platforms with strict banking standards.

You will build automated test frameworks, implement monitoring, and drive reliability initiatives across Azure/AWS environments. The ideal candidate has 7+ years of experience, strong Python expertise, and hands-on cloud

Qualifications

• 7+ years of relevant experience in AI testing, reliability engineering, or SRE roles.
• Proficient in Python and automation frameworks (PyTest, Selenium, Robot Framework).
• Hands-on with Azure or AWS networking, scaling, and serverless reliability.
• Familiarity with AI evaluation methods (RAG, ROUGE/ BLEU) and LLM benchmarks.

Responsibilities

• Design and execute automated testing frameworks for AI services such as Azure OpenAI and AWS Bedrock.
• Test AI solutions for model hallucinations, accuracy, content safety, and latency.
• Build CI/CD-integrated test suites using Python to validate application logic and infrastructure state.
• Implement Chaos Engineering and load testing for banking-scale web and mobile backends.
• Define and monitor SLIs and SLOs; configure monitoring with Azure Monitor or AWS CloudWatch.
• Automate security scans and align AI data handling with banking data residency and privacy requirements.

Skills

7+ years experience

Python

PyTest

Selenium

Robot Framework

Azure/AWS experience

AI evaluation concepts

LLM benchmarks

Monitoring tools

Bash

GitHub Actions

Terraform

K6/JMeter

DeepEval

Ragas/LangSmith

Tools

PyTest

Selenium

Robot Framework

Azure

AWS

Grafana

Prometheus

Azure Monitor

AWS CloudWatch

Bash

GitHub Actions

Terraform

K6

JMeter

DeepEval

Ragas

LangSmith

Job description

Role Overview

We are seeking for

Location:

Abu Dhabi, UAE

Experience:

7+ Years

Role Overview

We are seeking for AI Quality & Reliability Engineer to own the production readiness of cloud-based AI solutions.

This hybrid role combines automated software testing, AI model evaluation, and Site Reliability Engineering (SRE). The role focuses on building automated frameworks to validate AI outputs while ensuring Azure/AWS infrastructure is resilient, performant, and aligned with banking standards.

Key Responsibilities

AI Quality & Automated Testing

• Design and execute automated testing frameworks for AI services such as Azure OpenAI and AWS Bedrock.
• Test AI solutions for:
  • Model hallucinations
  • Accuracy
  • Content safety
  • Latency
• Build CI/CD-integrated test suites using Python to validate application logic and infrastructure state.

Reliability & Performance

• Implement Chaos Engineering and load testing for banking-scale web and mobile backends.
• Develop automated recovery scripts to support high availability.
• Define and monitor SLIs and SLOs.
• Configure advanced monitoring and alerting using Azure Monitor or AWS CloudWatch.

Security & Compliance

• Automate security scans and compliance checks.
• Ensure AI data handling aligns with banking data residency and privacy requirements.
• Identify inefficient cloud resource usage and expensive failing tests during testing.

Required Skills & Experience

• 7+ years of relevant experience.
• High proficiency in Python and automation frameworks such as:
  • PyTest
  • Selenium
  • Robot Framework
• Strong hands-on experience with Azure or AWS, particularly networking, scaling, and serverless reliability.
• Understanding of Prompt Engineering and AI output evaluation, including:
  • RAG evaluation
  • ROUGE/BLEU
  • Custom LLM benchmarks
• Experience with Grafana, Prometheus, Azure Monitor, AWS CloudWatch, or similar monitoring tools.
• Experience with Bash, GitHub Actions, Terraform, K6/JMeter.
• Exposure to AI evaluation tools such as DeepEval, Ragas, or LangSmith.