Senior AI QA Automation Architect

XM; Poland; On-site; PLN 180,000 - 280,000; Full time

Benefits offered by this job

Attractive remuneration package

Intellectually stimulating work env.

International training opportunities

Job summary

XM is seeking a Senior AI QA Test Automation Engineer to lead technical QA for AI-driven CX Agent Suite, focusing on evaluation pipelines and multi-agent interactions in AWS Bedrock context.

Collaborate with QA leadership, Data Science, and Engineering to architect scalable evaluation systems, prototype agent orchestration, and integrate testing into GitLab CI/CD.

The role emphasizes hands-on automation, data-driven quality analytics, mentoring, and influencing engineering standards across teams.

Qualifications

• 6+ years of hands-on QA/Test Automation experience.
• 1+ years applying AI/ML in software testing.
• Strong Python skills in AWS-centric environments; Java/TypeScript a plus.
• Hands-on experience with LLM evaluation techniques (LLM-as-a-judge, human-in-the-loop).
• Experience with DeepEval or similar evaluation frameworks; familiarity with agent-orchestration SDKs (Strands Agents).
• Knowledge of LLM tooling, vector databases, and MLOps pipelines.
• Experience integrating AI tooling into enterprise CI/CD (GitLab) and cloud-native environments (Docker, Kubernetes).
• Strong communicator, able to influence technical decisions across teams.

Responsibilities

• Design and build AI evaluation pipelines that assess multi-agent CX Agent Suite outputs for accuracy, relevance, tone, and safety.
• Develop scalable AI/QA frameworks and tooling for the CX Agent Suite and beyond.
• Evaluate and prototype Strands Agents or similar orchestration frameworks for self-evolving QA agents.
• Integrate AI-driven testing into the existing GitLab CI/CD pipeline and cloud-native environments.
• Lead research into AI testing methodologies and mentor engineers on code reviews and architectural guidance.

Skills

QA/Test Automation

Python

AI/ML in testing

CI/CD

Docker & Kubernetes

LLM evaluation

Strong communication

Education

BSc/MSc in Computer Science or AI

Tools

GitLab

Docker

Kubernetes

Terraform

OpenTelemetry

LangFuse

LangSmith

Arize

Job description

XM is seeking a Senior AI QA Test Automation Engineer to lead technical QA for AI-driven CX Agent Suite, focusing on evaluation pipelines and multi-agent interactions in AWS Bedrock context.

Collaborate with QA leadership, Data Science, and Engineering to architect scalable evaluation systems, prototype agent orchestration, and integrate testing into GitLab CI/CD.

The role emphasizes hands-on automation, data-driven quality analytics, mentoring, and influencing engineering standards across teams.