Senior QA Automation Engineer (Artificial Intelligence)

Anaplan; Greater London; Hybrid; GBP 90,000 - 120,000; Full time

Job summary

Anaplan is seeking a Senior SDET for AI to pioneer QA for GenAI/Agentic systems in London. You will design testing strategies and evaluation frameworks tailored to LLM-powered applications, and craft end-to-end automated tests in Python across API, UI, and performance domains.

You will also develop prompt testing systems, implement multi-dimensional GenAI quality metrics, and lead adversarial testing and production monitoring to safeguard reliability and safety.

Qualifications

• Deep understanding of GenAI technologies including LLMs, prompt engineering, and AI application patterns.
• Strong hands-on Python coding experience.
• Strong analytical and problem-solving skills with attention to detail.
• Strong experience with CI/CD pipelines and automated testing integration.
• Proven ability to design test cases and evaluation criteria for non-deterministic systems.
• Expert experience with test automation frameworks and scripting (Python, JavaScript, Selenium, Pytest).
• Experience with API testing tools (Postman, REST Assured) and backend testing.

Responsibilities

• Design and architect comprehensive test plans, evaluation frameworks, and quality metrics designed specifically for LLM-powered applications and agentic workflows.
• Develop robust, end-to-end automated test suites in Python across evaluations, API checks, UI checks, and performance tests.
• Create automated prompt testing systems and regression suites to proactively detect unintended changes in model behaviour.
• Establish multi-dimensional evaluation criteria to measure and track GenAI quality across accuracy, relevance, safety, consistency, and latency.
• Perform adversarial testing (red-teaming) to identify and mitigate model hallucinations, biases, or security vulnerabilities.
• Develop internal tools and testing frameworks that enable software engineering teams to seamlessly validate their own GenAI implementations.
• Deploy real-time monitoring and alerting systems to quickly detect and flag quality degradation of AI features in production.
• Own the overall quality and testing strategy for your workstream, maintaining high-fidelity test datasets (“golden examples”) and defining QA best practices.

Skills

Python

CI/CD

Test automation

Selenium

Pytest

API testing

Postman

REST Assured

GenAI

LLMs

Prompt engineering

QA testing

Adversarial testing

Tools

Selenium

Postman

REST Assured

Pytest

Job description

• We’re pioneering a new role focused exclusively on quality assurance for GenAI/Agentic systems. As our Senior SDET for AI, you’ll develop testing strategies, evaluation frameworks, and quality metrics specifically designed for LLM-powered applications. This role requires a unique blend of QA expertise, understanding of GenAI behaviour, and automation skills to ensure our AI features are reliable, accurate, and trustworthy
• Design and architect comprehensive test plans, evaluation frameworks, and quality metrics designed specifically for LLM-powered applications and agentic workflows
• Develop robust, end-to-end automated test suites in Python across evaluations, API checks, UI checks, and performance tests
• Create automated prompt testing systems and regression suites to proactively detect unintended changes in model behaviour
• Establish multi-dimensional evaluation criteria to measure and track GenAI quality across accuracy, relevance, safety, consistency, and latency
• Perform adversarial testing (red-teaming) to identify and mitigate model hallucinations, biases, or security vulnerabilities
• Develop internal tools and testing frameworks that enable software engineering teams to seamlessly validate their own GenAI implementations
• Deploy real-time monitoring and alerting systems to quickly detect and flag quality degradation of AI features in production
• Own the overall quality and testing strategy for your workstream, maintaining high-fidelity test datasets (“golden examples”) and defining QA best practices

Deep understanding of GenAI technologies including LLMs, prompt engineering, and AI application patternsStrong hands-on Python coding experienceStrong analytical and problem-solving skills with attention to detailStrong experience with CI/CD pipelines and automated testing integrationStrong background of software testing methodologies (functional, integration, regression, performance, security testing)Proven ability to design test cases and evaluation criteria for non-deterministic systemsExpert experience with test automation frameworks and scripting (Python, JavaScript, Selenium, Pytest)Experience with API testing tools (Postman, REST Assured) and backend testing