AI Quality & Reliability Engineer (QA/SRE)

Dicetek LLC; Dubai; On-site; AED 201,000 - 290,000; Full time

Job summary

Dicetek LLC is seeking a technically skilled automation tester in Dubai to drive AI evaluation and cloud-enabled testing, leveraging Python, PyTest, and Selenium within CI/CD pipelines. You will optimize tests for cost and reliability, and work with Terraform-managed infrastructure, K6/JMeter for performance, and AI evaluation tools like LangSmith or DeepEval.

The role emphasizes hands-on cloud experience, monitoring setup with Grafana/Prometheus, and a proactive FinOps mindset to reduce waste

Qualifications

• High proficiency in Python for AI testing and framework automation (PyTest, Selenium, or Robot Framework).
• Cloud infrastructure experience with Azure or AWS, especially networking, scaling, and serverless reliability.
• Understanding of AI/ML evaluation (RAG evaluation, ROUGE/ BLEU scores, or custom LLM benchmarks).
• Experience with real-time monitoring tools to build reliability dashboards (Grafana, Prometheus, or native cloud tools).
• FinOps awareness to identify expensive tests or inefficient cloud resource usage during testing.

Skills

Python

Bash scripting

GitHub Actions

Terraform

K6

JMeter

PyTest

Selenium

Robot Framework

Azure

AWS

Grafana

Prometheus

RAG evaluation

LangSmith

DeepEval

Tools

GitHub Actions (CI/CD)

Terraform

K6

JMeter

Job description

Technical & Professional Requirements:

Automation Stack: High proficiency in Python (for AI testing) and framework automation (PyTest, Selenium, or Robot Framework).

Cloud Infrastructure: Strong hands-on experience with Azure or AWS, specifically regarding networking, scaling, and serverless reliability.

AI/ML Understanding: Understanding of Prompt Engineering and how to evaluate AI model outputs (RAG evaluation, ROUGE/BLEU scores, or custom LLM-benchmarks).

Monitoring Tools: Experience with Grafana, Prometheus, or native cloud monitoring tools to build real-time reliability dashboards.

FinOps Awareness: Ability to identify "expensive" failing tests or inefficient cloud resource usage during the testing phase.

Recommended Skillset & Tools:

Languages: Python (Mandatory), Bash scripting.

Tools: GitHub Actions (CI/CD), Terraform (reading/validating), K6 or JMeter (Performance).

AI Frameworks: DeepEval, Ragas, or LangSmith (for automated AI evaluation).