AI Quality & Reliability Engineer – QA/SRE

D4Insight; Abu Dhabi; Hybrid; AED 280,000 - 420,000; Full time

Job summary

D4Insight in Abu Dhabi is looking for an AI Quality & Reliability Engineer to own the production readiness of cloud-based AI solutions. This hybrid role blends automated testing, AI model evaluation, and SRE, focusing on frameworks that validate AI outputs while ensuring Azure/AWS infrastructure is resilient and bank-grade compliant.

Responsibilities include designing CI/CD test suites, testing AI services for hallucinations, accuracy, and latency, and implementing chaos engineering with robust

Qualifications

• 7+ years of relevant experience in QA/SRE for AI or cloud apps.
• Strong Python expertise with automation frameworks (PyTest, Selenium, Robot Framework).
• Hands-on Azure or AWS, including networking, scaling, and serverless reliability.
• Experience with AI evaluation: RAG, ROUGE/BLEU, and LLM benchmarks.
• Proficient in monitoring tools: Grafana/Prometheus, Azure Monitor, AWS CloudWatch.
• Experience with CI/CD, Terraform, Bash, and GitHub Actions.

Responsibilities

• Design and execute automated testing frameworks for AI services.
• Test AI solutions for hallucinations, accuracy, content safety, and latency.
• Build CI/CD-integrated test suites to validate logic and infra state.
• Implement Chaos Engineering and load testing for banking-scale backends.
• Develop automated recovery scripts and define SLIs/SLOs.
• Automate security scans and ensure banking data residency and privacy.

Skills

Python

PyTest

Selenium

Robot Framework

Azure

AWS

CI/CD

Terraform

K6

JMeter

Grafana

Prometheus

RAG evaluation

LangSmith

DeepEval

Tools

Azure Monitor

AWS CloudWatch

GitHub Actions

Terraform

Bash

Job description

AI Quality & Reliability Engineer – QA/SRE

Role Overview:

We are seeking for AI Quality & Reliability Engineer to own the production readiness of cloud-based AI solutions.

This hybrid role combines automated software testing, AI model evaluation, and Site Reliability Engineering (SRE). The role focuses on building automated frameworks to validate AI outputs while ensuring Azure/AWS infrastructure is resilient, performant, and aligned with banking standards.

Key Responsibilities:

AI Quality & Automated Testing

• Design and execute automated testing frameworks for AI services such as Azure OpenAI and AWS Bedrock.
• Test AI solutions for:
  • Model hallucinations
  • Accuracy
  • Content safety
  • Latency
• Build CI/CD-integrated test suites using Python to validate application logic and infrastructure state.

Reliability & Performance

• Implement Chaos Engineering and load testing for banking-scale web and mobile backends.
• Develop automated recovery scripts to support high availability.
• Define and monitor SLIs and SLOs.
• Configure advanced monitoring and alerting using Azure Monitor or AWS CloudWatch.

Security & Compliance

• Automate security scans and compliance checks.
• Ensure AI data handling aligns with banking data residency and privacy requirements.
• Identify inefficient cloud resource usage and expensive failing tests during testing.

Required Skills & Experience:

• 7+ years of relevant experience.
• High proficiency in Python and automation frameworks such as:
  • PyTest
  • Selenium
  • Robot Framework
• Strong hands-on experience with Azure or AWS, particularly networking, scaling, and serverless reliability.
• Understanding of Prompt Engineering and AI output evaluation, including:
  • RAG evaluation
  • ROUGE/BLEU
  • Custom LLM benchmarks
• Experience with Grafana, Prometheus, Azure Monitor, AWS CloudWatch, or similar monitoring tools.
• Experience with Bash, GitHub Actions, Terraform, K6/JMeter.
• Exposure to AI evaluation tools such as DeepEval, Ragas, or LangSmith.

We craft, deploy, and manage bespoke services in CRM, data and AI, cybersecurity and consulting.