GenAI QA Automation Engineer

Anaplan; Greater London; On-site; GBP 70,000 - 100,000; Full time

Job summary

Anaplan is hiring a QA Automation Engineer focused on GenAI/Agentic systems to build and run automated tests across evaluations, APIs, UIs, and performance. You will design evaluation frameworks and quality metrics for LLM-powered features, ensuring reliability, accuracy and safety in AI workflows.

You will work within a team shaping testing strategies for conversational AI and production-grade GenAI applications, leveraging Python-based tooling and PyTest to automate validation and regression

Qualifications

• Solid QA or test engineering experience, with exposure to AI/ML systems.
• Hands on Python coding experience.
• Good understanding of GenAI technologies including LLMs, prompt engineering, and AI application patterns.
• Solid experience with test automation frameworks and scripting (Python, Pytest, API, and UI Testing).
• Practical knowledge of software testing methodologies (functional, integration, regression, performance, security testing).
• Ability to execute test cases and evaluation criteria for non-deterministic systems.
• Experience with API testing tools (Postman, REST Assured) and backend testing.
• Familiarity with CI/CD pipelines and automated testing integration.

Responsibilities

• Develop and execute end-to-end automated test coverage across evaluations, APIs, UIs, and performance tests using Python.
• Implement testing strategies for GenAI features, including conversational AI, agentic systems, and LLM-powered workflows within your workstream.
• Run and maintain automated test suites for prompt testing, including regression tests designed to detect unintended changes in model behaviour.
• Support and maintain evaluation frameworks to measure GenAI performance across accuracy, relevance, safety, consistency, and latency.
• Contribute to curating high-quality test datasets and 'golden examples' that represent diverse user scenarios and edge cases.
• Assist in adversarial testing to identify potential model hallucinations or biases, and support the integration of production alerting systems.
• Utilise internal tools and testing frameworks to help developers easily validate their GenAI code, and assist with user acceptance testing.
• Take accountability for your workstream's automated coverage, documenting testing procedures and known issues in clear, accessible formats.

Skills

QA automation

Python coding

GenAI testing

API testing

UI testing

CI/CD integration

Non-deterministic testing

Tools

Pytest

Postman

REST Assured

Job description

Anaplan is hiring a QA Automation Engineer focused on GenAI/Agentic systems to build and run automated tests across evaluations, APIs, UIs, and performance. You will design evaluation frameworks and quality metrics for LLM-powered features, ensuring reliability, accuracy and safety in AI workflows.

You will work within a team shaping testing strategies for conversational AI and production-grade GenAI applications, leveraging Python-based tooling and PyTest to automate validation and regression