AI Quality & Reliability Engineer (QA/SRE)

Dicetek LLC; Abu Dhabi; Hybrid; AED 180,000 - 260,000; Full time

Job summary

Tanqeeb in the United Arab Emirates seeks an AI Quality & Reliability Engineer to own Production Readiness of cloud-based AI solutions in a hybrid role, blending automated testing, AI model evaluation, and SRE.

You will design AI quality automation, implement chaos testing, CI/CD-integrated test suites in Python, define SLIs/SLOs, and perform security checks for strict banking data residency and privacy requirements.

Qualifications

• Automation Stack: Python and framework automation (PyTest, Selenium, or Robot Framework).
• Cloud Infrastructure: Hands-on with Azure or AWS networking, scaling, and serverless reliability.
• AI/ML Understanding: Prompt engineering and AI model output evaluation (ROUGE/BLEU or custom benchmarks).
• Monitoring Tools: Grafana, Prometheus, or native cloud monitoring for real-time dashboards.
• FinOps Awareness: Identify expensive failing tests or inefficient cloud usage during testing.

Responsibilities

• AI Quality Automation: Design and execute automated testing for AI services (Azure OpenAI, AWS Bedrock).
• Resiliency Engineering (SRE): Implement Chaos Engineering and load testing for high-traffic backends; ensure automated recovery scripts.

Skills

Python

Azure

AWS

AI model evaluation

CI/CD automation

Monitoring/Observability

Tools

GitHub Actions

Terraform

K6

JMeter

Grafana

Prometheus

Job description

We are seeking an AI Quality & Reliability Engineer to own the \"Production Readiness\" of our cloud-based AI solutions. This hybrid role combines automated software testing, AI model evaluation, and Site Reliability Engineering (SRE).

Key Responsibilities

• AI Quality Automation: Design and execute automated testing frameworks for AI services (e.g., Azure OpenAI, AWS Bedrock). This includes testing for model hallucinations, accuracy, and \"AI Content Safety\" latency.




• Resiliency Engineering (SRE): Implement \"Chaos Engineering\" and load testing to ensure web/mobile backends can handle banking-scale traffic. Maintain high availability through automated recovery scripts.




• Automated Regression: Build CI/CD-integrated test suites using Python that validate both the application logic and the infrastructure state (IaC validation).




• Observability & SLIs: Define and monitor Service Level Indicators (SLIs) and Objectives (SLOs). Set up advanced alerting in Azure Monitor or AWS CloudWatch to catch performance degradation before users do.




• Security & Compliance Testing: Automate security scans and compliance checks to ensure all AI data handling meets strict banking data residency and privacy protocols.

Desired Candidate Profile

Technical & Professional Requirements:

• Automation Stack: High proficiency in Python (for AI testing) and framework automation (PyTest, Selenium, or Robot Framework).




• Cloud Infrastructure: Strong hands-on experience with Azure or AWS, specifically regarding networking, scaling, and serverless reliability.




• AI/ML Understanding: Understanding of Prompt Engineering and how to evaluate AI model outputs (RAG evaluation, ROUGE/BLEU scores, or custom LLM-benchmarks).




• Monitoring Tools: Experience with Grafana, Prometheus, or native cloud monitoring tools to build real-time reliability dashboards.




• FinOps Awareness: Ability to identify \"expensive\" failing tests or inefficient cloud resource usage during the testing phase.

Recommended Skillset & Tools

• Languages: Python (Mandatory), Bash scripting.




• Tools: GitHub Actions (CI/CD), Terraform (reading/validating), K6 or JMeter (Performance).




• AI Frameworks: DeepEval, Ragas, or LangSmith (for automated AI evaluation).

Skills

Python

Azure/AWS

AI model evaluation