Automation Test Engineer (AI & Chatbot Testing)

DXC Technology; Dubai; On-site; AED 180,000 - 240,000; Full time

Job summary

DXC Technology in Dubai is seeking an experienced Automation Test Engineer to validate AI-powered customer support and troubleshooting solutions, focusing on test automation, API testing, and AI quality validation.

You will measure AI performance, build automated tests for chatbots, integrate tests into Azure DevOps pipelines, and collaborate with AI engineers and product teams to ensure reliable, scalable testing across platforms.

Qualifications

• 4+ years of QA Automation or SDET experience.
• 1+ year of experience testing AI, Chatbots, LLMs, or Agentic AI solutions.
• Strong programming skills in Python, TypeScript, or C#.
• Experience with REST APIs, GraphQL, WebSockets, Postman, and WireMock.
• Experience with Azure DevOps and CI/CD pipelines.
• Knowledge of RAG, semantic search, and vector databases.
• Experience with MQTT or IoT telemetry is a plus.
• Exposure to embedded systems or consumer electronics is preferred.

Responsibilities

• Develop automated tests for AI chatbots and RAG systems.
• Validate response accuracy, relevance, and AI performance.
• Measure AI quality metrics such as hallucination rates and answer relevance.
• Perform regression testing when AI models or knowledge bases updated.
• Automate testing of troubleshooting workflows and decision trees.
• Simulate customer interactions and verify expected outcomes.
• Build and execute automated API tests.
• Validate integrations between AI platforms and backend systems.
• Create mock services and test data where required.
• Conduct load and performance testing of conversational AI systems.
• Test AI guardrails, prompt injection risks, and unsafe responses.
• Identify performance and reliability issues.
• Integrate automated tests into Azure DevOps pipelines.
• Track quality metrics and generate test reports.Work closely with developers, AI engineers, and product teams.

Skills

Python

TypeScript

C#

REST APIs

GraphQL

WebSockets

Postman

WireMock

Azure DevOps

CI/CD pipelines

RAG

Semantic search

Vector databases

MQTT

Embedded systems

Tools

Postman

WireMock

Azure DevOps

Job description

Automation Test Engineer (AI & Chatbot Testing)

Location: UAE

Job Summary

We are looking for an experienced Automation Test Engineer to test and improve AI-powered customer support and troubleshooting solutions. The role focuses on test automation, API testing, AI quality validation, and performance testing to ensure reliable and accurate customer experiences.

Key Responsibilities

AI & Chatbot Testing

• Develop automated tests for AI chatbots and RAG (Retrieval-Augmented Generation) systems.
• Validate response accuracy, relevance, and AI performance.
• Measure AI quality metrics such as hallucination rates and answer relevance.
• Perform regression testing when AI models or knowledge bases updated.

Automation & Functional Testing

• Automate testing of troubleshooting workflows and decision trees.
• Simulate customer interactions and verify expected outcomes.

API & Integration Testing

• Build and execute automated API tests.
• Validate integrations between AI platforms and backend systems.
• Create mock services and test data where required.

Performance & Security Testing

• Conduct load and performance testing of conversational AI systems.
• Test AI guardrails, prompt injection risks, and unsafe responses.
• Identify performance and reliability issues.

CI/CD & Quality Assurance

• Integrate automated tests into Azure DevOps pipelines.
• Track quality metrics and generate test reports.Work closely with developers, AI engineers, and product teams.

Requirements

• 4+ years of experience in QA Automation or SDET roles.
• 1+ year of experience testing AI, Chatbots, LLMs, or Agentic AI solutions.
• Strong programming skills in Python, TypeScript, or C#.
• Experience with REST APIs, GraphQL, WebSockets, Postman, and WireMock.
• Experience with Azure DevOps and CI/CD pipelines.
• Knowledge of RAG, semantic search, and vector databases.
• Experience with MQTT or IoT telemetry is a plus.
• Exposure to embedded systems or consumer electronics is preferred.