AI Quality & Reliability Engineer: Cloud, AI Testing & Monitoring

Dicetek LLC; Dubai; On-site; AED 201,000 - 290,000; Full time

Job summary

Dicetek LLC is seeking a technically skilled automation tester in Dubai to drive AI evaluation and cloud-enabled testing, leveraging Python, PyTest, and Selenium within CI/CD pipelines. You will optimize tests for cost and reliability, and work with Terraform-managed infrastructure, K6/JMeter for performance, and AI evaluation tools like LangSmith or DeepEval.

The role emphasizes hands-on cloud experience, monitoring setup with Grafana/Prometheus, and a proactive FinOps mindset to reduce waste

Qualifications

• High proficiency in Python for AI testing and framework automation (PyTest, Selenium, or Robot Framework).
• Cloud infrastructure experience with Azure or AWS, especially networking, scaling, and serverless reliability.
• Understanding of AI/ML evaluation (RAG evaluation, ROUGE/ BLEU scores, or custom LLM benchmarks).
• Experience with real-time monitoring tools to build reliability dashboards (Grafana, Prometheus, or native cloud tools).
• FinOps awareness to identify expensive tests or inefficient cloud resource usage during testing.

Skills

Python

Bash scripting

GitHub Actions

Terraform

K6

JMeter

PyTest

Selenium

Robot Framework

Azure

AWS

Grafana

Prometheus

RAG evaluation

LangSmith

DeepEval

Tools

GitHub Actions (CI/CD)

Terraform

K6

JMeter

Job description

Dicetek LLC is seeking a technically skilled automation tester in Dubai to drive AI evaluation and cloud-enabled testing, leveraging Python, PyTest, and Selenium within CI/CD pipelines. You will optimize tests for cost and reliability, and work with Terraform-managed infrastructure, K6/JMeter for performance, and AI evaluation tools like LangSmith or DeepEval.

The role emphasizes hands-on cloud experience, monitoring setup with Grafana/Prometheus, and a proactive FinOps mindset to reduce waste