AI Quality & Reliability Engineer (QA/SRE) | Dicetek | UAE

Dicetek; United Arab Emirates; On-site; AED 300,000 - 540,000; Full time

Job summary

Dicetek is seeking an experienced AI Quality & Reliability Engineer to lead automated testing, RAG evaluation, and LLM benchmarking in the UAE. You will own end-to-end testing and reliability for distributed AI platforms, integrating PyTest, Terraform, Prometheus, and Grafana across cloud environments.

You will collaborate with AI platform engineers, MLOps and infra teams to ensure low-latency AI workloads, scalable testing, and robust cost controls while advancing CI/CD with GitHub Actions.

Qualifications

• Bachelor’s degree in Computer Science, Software Engineering, Artificial Intelligence, or a related technical discipline.
• 4+ years of professional experience in software quality assurance, site reliability engineering, or AI systems validation.
• High proficiency in Python for AI testing automation, scripting, and framework development (PyTest, Selenium, or Robot Framework).
• Strong hands-on experience with cloud infrastructure platforms (Azure or AWS).
• Practical understanding of prompt engineering, RAG evaluation, and LLM benchmarking methodologies.

Responsibilities

• Design, develop, and execute automated testing frameworks for AI/ML systems and cloud-native microservices using Python.
• Evaluate AI model outputs, retrieval-augmented generation pipelines, and LLM benchmarks using DeepEval, Ragas, or LangSmith.
• Measure and track generation quality metrics including ROUGE, BLEU, and custom evaluation scores.
• Configure, manage, and monitor real-time reliability dashboards and alerting systems using Prometheus and Grafana.
• Build and maintain automated CI/CD deployment pipelines and testing workflows utilizing GitHub Actions.
• Validate and review infrastructure-as-code provisioning templates and modules written in Terraform.
• Conduct rigorous performance, load, and stress testing for AI APIs using K6 or JMeter.
• Implement FinOps cost-control measures to identify expensive failing tests and optimize cloud resource usage.
• Collaborate with AI engineers, MLOps, cybersecurity, and infrastructure teams in the UAE.

Skills

Python for AI testing

PyTest

SRE/QA for AI systems

Cloud infrastructure

LLM benchmarking

RAG evaluation

Observability

Education

Bachelor's degree in Computer Science/Software Engineering/AI

Tools

Azure

AWS

Terraform

GitHub Actions

Prometheus

Grafana

K6

JMeter

LangSmith/DeepEval/Ragas

Job description

Position Summary

Dicetek is seeking an experienced, highly technical, and meticulous AI Quality & Reliability Engineer (QA/SRE) to join our technology team in the UAE. In this specialized artificial intelligence quality assurance and site reliability role, you will spearhead automated testing, RAG evaluation, LLM benchmarking, and cloud-native reliability engineering for advanced AI systems. You will work closely with cross-functional AI platform engineers, MLOps specialists, and infrastructure teams to ensure high availability, optimal prompt performance, and robust FinOps cost control across production AI workflows. Ideal candidates bring a strong academic foundation in computer science, deep mastery of Python and cloud infrastructure, and a proven track record of validating scalable AI solutions on Azure or AWS.

Detailed Job Description

As an AI Quality & Reliability Engineer at Dicetek in the UAE, you will take full ownership of building, executing, and maintaining comprehensive testing and reliability frameworks for large language models and distributed AI applications. Your day-to-day responsibilities encompass developing automated test suites in Python using PyTest, evaluating AI model outputs with modern assessment tools like DeepEval, Ragas, or LangSmith, and monitoring cloud infrastructure health utilizing Prometheus and Grafana. You will manage CI/CD pipelines via GitHub Actions, validate infrastructure-as-code scripts in Terraform, and conduct performance testing using K6 or JMeter. Working in a fast-paced environment, you will identify inefficient cloud resource usage, optimize FinOps practices, and guarantee low-latency reliability across all deployed AI platforms.

Key Responsibilities

• Design, develop, and execute automated testing frameworks for AI/ML systems and cloud-native microservices using Python.
• Evaluate AI model outputs, retrieval-augmented generation (RAG) pipelines, and LLM benchmarks using DeepEval, Ragas, or LangSmith.
• Measure and track generation quality metrics including ROUGE, BLEU, and custom domain-specific evaluation scores.
• Configure, manage, and monitor real-time reliability dashboards and alerting systems using Prometheus and Grafana.
• Build and maintain automated CI/CD deployment pipelines and testing workflows utilizing GitHub Actions.
• Validate and review infrastructure-as-code provisioning templates and modules written in Terraform.
• Conduct rigorous performance, load, and stress testing for AI APIs using K6 or JMeter.
• Implement FinOps cost-control measures to identify expensive failing tests and optimize cloud resource usage.
• Collaborate closely with AI engineers, MLOps specialists, cybersecurity, and infrastructure teams in the UAE.

Required Qualifications & Skills

• Bachelor’s degree in Computer Science, Software Engineering, Artificial Intelligence, or a related technical discipline.
• 4+ years of professional experience in software quality assurance, site reliability engineering, or AI systems validation.
• High proficiency in Python for AI testing automation, scripting, and framework development (PyTest, Selenium, or Robot Framework).
• Strong hands-on experience with cloud infrastructure platforms (Azure or AWS), focusing on networking, scaling, and serverless reliability.
• Practical understanding of prompt engineering, RAG evaluation frameworks, and LLM benchmarking methodologies.
• Extensive experience with monitoring and observability tools including Grafana, Prometheus, or native cloud suites.
• Solid awareness of FinOps principles and cloud resource cost optimization during testing phases.
• Professional availability for full-time deployment in the UAE.

Nice-to-Have Skills

• Professional certifications such as AWS Certified DevOps Engineer, Microsoft Certified: Azure Administrator, or ISTQB Agile Tester.
• Advanced experience with specialized AI evaluation frameworks including LangSmith, DeepEval, or Ragas.
• Familiarity with containerization and orchestration platforms such as Docker and Kubernetes.
• Prior working experience in generative AI startups, enterprise SaaS, or digital transformation technology sectors.
• Contributions to open-source AI testing libraries or active participation in machine learning reliability communities.

Salary & Contract

• Salary/Rate: Competitive and commensurate with experience
• Contract Duration: Permanent / Full-Time
• Location: UAE