Senior QA Engineer – Azure AI

Talenzon; Doha; On-site; QAR 254,823 - 327,629; Full time

Job summary

Talenzon is looking for a highly skilled Senior QA Engineer specializing in Azure AI to join their team in Doha. The role requires expertise in evaluating AI systems, robust experience in Azure, and advanced Python skills.

In this full-time position, you will design QA strategies, build automated pipelines, and ensure the reliability of AI implementations. Candidates should be proficient in Generative AI and have a strong QA background.

Qualifications

• 1-2 years of hands-on experience with Generative AI.
• 5+ years of experience using Azure platforms.
• 7+ years of experience coding in Python, including frameworks.
• 8+ years of QA experience with a focus on LLM evaluation metrics.
• 5+ years working with DataOps or CI/CD pipelines.

Responsibilities

• Design and execute QA strategies for AI systems.
• Build automated evaluation pipelines in Azure DevOps.
• Develop testing frameworks using Python.
• Test and validate AI deployments across Azure services.
• Contribute to DataOps processes for performance monitoring.

Skills

Generative AI experience

Azure expertise

Python coding

QA and Automation

DataOps / CI/CD knowledge

Education

Technical degree in Computer Science, Engineering, Mathematics or equivalent

Tools

Azure DevOps

Pandas

PyTest

MLflow

Job description

We are seeking a highly skilled Senior QA Engineer specializing in Azure AI to join our client’s team in Doha (on-site). This role focuses on ensuring the reliability, accuracy, and safety of AI/LLM systems deployed on Azure, with a strong emphasis on evaluation frameworks, automation, and modern GenAI architectures.

Location: Doha, Qatar — On-site
Role Type: Full‑Time, Permanent
Language Required: English

Responsibilities

• Design and execute QA strategies for LLM, Generative AI, and RAG‑based systems, ensuring reliability, correctness, and stability.
• Build automated evaluation pipelines using Azure DevOps, integrating model testing into CI/CD workflows.
• Develop advanced Python‑based testing frameworks, including Pandas workflows, PyTest suites, and custom evaluation scripts.
• Validate AI behaviours using both deterministic assertions and LLM‑as‑a‑Judge evaluation methodologies.
• Collaborate with data engineering and platform teams to maintain evaluation datasets, including ground truth / golden datasets.
• Test and validate Azure AI deployments end‑to‑end, including Azure AI Studio, Azure OpenAI, Azure AI Search, and related services.
• Contribute to DataOps processes and continuous monitoring of AI system performance, regression, and drift.
• Support experimentation workflows, ensuring reproducibility, traceability, and quality across multiple model versions.
• Stay updated on emerging GenAI methodologies, including agentic architectures, multi‑agent workflows, and evaluation best practices.

Required Qualifications

Generative AI

• 1–2 years of hands‑on Generative AI experience.
• Practical expertise with RAG architectures (Vector Search + LLM).

Azure Cloud

• 5+ years of Azure experience.
• Proven deployment/testing experience in Azure AI Studio or Azure OpenAI Service (not just local testing).

Coding – Python

• 7+ years of Python experience.
• Ability to write complex, production‑quality code (Pandas, PyTest, automation frameworks).

QA & Automation

• 8+ years of QA experience.
• Strong understanding of LLM evaluation metrics and frameworks.

DataOps / CI/CD

• 5+ years working with DataOps or CI/CD pipelines.
• Direct experience with Azure DevOps pipelines tied to AI evaluations.

Education

• Technical degree in Computer Science, Engineering, Mathematics, or equivalent professional experience.

Preferred Qualifications (Should Have)

Generative AI

• Experience with Agentic AI (LangGraph, AutoGen) or multi-agent systems.

Azure Cloud

• Experience with Azure AI Search / Cognitive Search including indexing and vector optimization.

Coding – Python

• Experience with LangChain or Semantic Kernel.

QA & Automation

• Experience building evaluation pipelines using Prompt Flow.

DataOps / CI/CD

• Experience creating and managing Golden Datasets.

Education

• Master’s degree in Data Science or AI‑related field.

Nice to Have

Generative AI

• Experience fine‑tuning SLMs (e.g., Phi‑3, Llama).

Azure Cloud

• Familiarity with Azure ML, Workspaces, and Compute management.

Coding – Python

• Background in C#/.NET for backend integration.

QA & Automation

• Exposure to Red Teaming and adversarial testing.

DataOps / CI/CD

• Experience with MLflow.

Education

• Certification: Azure AI Engineer Associate (AI‑102).