AI Quality & Reliability Engineer (QA/SRE)

Dicetek LLC; Abu Dhabi; Hybrid; AED 180,000 - 300,000; Full time

Job summary

Dicetek LLC is seeking an AI Quality & Reliability Engineer to own production readiness of cloud-based AI solutions. This hybrid role combines automated testing, AI model evaluation, and SRE for Azure/AWS infrastructure in a banking context.

You will build frameworks that validate AI outputs and ensure resilient, compliant operations, with a strong focus on monitoring, security testing, and cost-conscious testing practices.

Qualifications

• Automation stack with Python for AI testing and framework automation (PyTest, Selenium, or Robot Framework).
• Cloud infrastructure experience with Azure or AWS focusing on networking, scaling, and serverless reliability.
• Understanding of AI/ML evaluation methods such as prompt engineering and LLM benchmarks.
• Monitoring and observability proficiency with Grafana/Prometheus or native cloud tools for real-time dashboards.
• FinOps awareness to identify expensive tests or inefficient cloud usage during testing.

Responsibilities

• AI Quality Automation: design and run automated testing for AI services, incl. model evaluation and content safety latency.
• Resiliency Engineering (SRE): implement chaos engineering and load testing, ensure high availability with automated recovery scripts.
• Automated Regression: build CI/CD-integrated test suites validating app logic and IaC state.
• Observability & SLIs: define and monitor SLIs/SLOs; set up alerts in Azure Monitor or CloudWatch to catch degradation.
• Security & Compliance Testing: automate security scans and compliance checks for banking data residency and privacy requirements.

Skills

Python

PyTest

Selenium

Robot Framework

Azure

AWS

Terraform

Grafana

Prometheus

CI/CD

Tools

GitHub Actions

Terraform

K6

JMeter

Grafana

Prometheus

Job description

Role Overview

We are seeking an AI Quality & Reliability Engineer to own the "Production Readiness" of our cloud-based AI solutions. This hybrid role combines automated software testing, AI model evaluation, and Site Reliability Engineering (SRE). You will build the automated frameworks that validate our AI outputs and ensure the underlying Azure/AWS infrastructure is resilient, performant, and compliant with banking standards.

Key Responsibilities

• AI Quality Automation: Design and execute automated testing frameworks for AI services (e.g., Azure OpenAI, AWS Bedrock). This includes testing for model hallucinations, accuracy, and "AI Content Safety" latency.
• Resiliency Engineering (SRE): Implement "Chaos Engineering" and load testing to ensure web/mobile backends can handle banking-scale traffic. Maintain high availability through automated recovery scripts.
• Automated Regression: Build CI/CD-integrated test suites using Python that validate both the application logic and the infrastructure state (IaC validation).
• Observability & SLIs: Define and monitor Service Level Indicators (SLIs) and Objectives (SLOs). Set up advanced alerting in Azure Monitor or AWS CloudWatch to catch performance degradation before users do.
• Security & Compliance Testing: Automate security scans and compliance checks to ensure all AI data handling meets strict banking data residency and privacy protocols.

Technical & Professional Requirements

• Automation Stack: High proficiency in Python (for AI testing) and framework automation (PyTest, Selenium, or Robot Framework).
• Cloud Infrastructure: Strong hands-on experience with Azure or AWS, specifically regarding networking, scaling, and serverless reliability.
• AI/ML Understanding: Understanding of Prompt Engineering and how to evaluate AI model outputs (RAG evaluation, ROUGE/BLEU scores, or custom LLM-benchmarks).
• Monitoring Tools: Experience with Grafana, Prometheus, or native cloud monitoring tools to build real-time reliability dashboards.
• FinOps Awareness: Ability to identify "expensive" failing tests or inefficient cloud resource usage during the testing phase.

Recommended Skillset & Tools

• Languages: Python (Mandatory), Bash scripting.
• Tools: GitHub Actions (CI/CD), Terraform (reading/validating), K6 or JMeter (Performance).
• AI Frameworks: DeepEval, Ragas, or LangSmith (for automated AI evaluation).