AI Quality & Reliability Engineer (QA/SRE)

Dicetek; United Arab Emirates; On-site; AED 300,000 - 540,000; Full time

Job summary

Dicetek is seeking an experienced AI Quality & Reliability Engineer to lead automated testing, RAG evaluation, and LLM benchmarking in the UAE. You will own end-to-end testing and reliability for distributed AI platforms, integrating PyTest, Terraform, Prometheus, and Grafana across cloud environments.

You will collaborate with AI platform engineers, MLOps and infra teams to ensure low-latency AI workloads, scalable testing, and robust cost controls while advancing CI/CD with GitHub Actions.

Qualifications

• Bachelor’s degree in Computer Science, Software Engineering, Artificial Intelligence, or a related technical discipline.
• 4+ years of professional experience in software quality assurance, site reliability engineering, or AI systems validation.
• High proficiency in Python for AI testing automation, scripting, and framework development (PyTest, Selenium, or Robot Framework).
• Strong hands-on experience with cloud infrastructure platforms (Azure or AWS).
• Practical understanding of prompt engineering, RAG evaluation, and LLM benchmarking methodologies.

Responsibilities

• Design, develop, and execute automated testing frameworks for AI/ML systems and cloud-native microservices using Python.
• Evaluate AI model outputs, retrieval-augmented generation pipelines, and LLM benchmarks using DeepEval, Ragas, or LangSmith.
• Measure and track generation quality metrics including ROUGE, BLEU, and custom evaluation scores.
• Configure, manage, and monitor real-time reliability dashboards and alerting systems using Prometheus and Grafana.
• Build and maintain automated CI/CD deployment pipelines and testing workflows utilizing GitHub Actions.
• Validate and review infrastructure-as-code provisioning templates and modules written in Terraform.
• Conduct rigorous performance, load, and stress testing for AI APIs using K6 or JMeter.
• Implement FinOps cost-control measures to identify expensive failing tests and optimize cloud resource usage.
• Collaborate with AI engineers, MLOps, cybersecurity, and infrastructure teams in the UAE.

Skills

Python for AI testing

PyTest

SRE/QA for AI systems

Cloud infrastructure

LLM benchmarking

RAG evaluation

Observability

Education

Bachelor's degree in Computer Science/Software Engineering/AI

Tools

Azure

AWS

Terraform

GitHub Actions

Prometheus

Grafana

K6

JMeter

LangSmith/DeepEval/Ragas

Job description

Dicetek is seeking an experienced AI Quality & Reliability Engineer to lead automated testing, RAG evaluation, and LLM benchmarking in the UAE. You will own end-to-end testing and reliability for distributed AI platforms, integrating PyTest, Terraform, Prometheus, and Grafana across cloud environments.

You will collaborate with AI platform engineers, MLOps and infra teams to ensure low-latency AI workloads, scalable testing, and robust cost controls while advancing CI/CD with GitHub Actions.