GenAI QA Automation Engineer – LLM Testing & Automation

Anaplan; Greater London; On-site; GBP 70,000 - 110,000; Full time

Job summary

Anaplan is pioneering a new QA automation role focused on GenAI/Agentic systems. As a QA Automation Engineer within AI, you'll design testing strategies, evaluation frameworks, and quality metrics for LLM-powered features to ensure reliability and trustworthiness.

You'll develop end-to-end automated test coverage across evaluations, APIs, UIs, and performance, implement GenAI test strategies, run regression tests, and help maintain datasets and golden examples for diverse scenarios and edge

Qualifications

• Experience in QA/testing for AI/ML or GenAI systems.
• Strong Python skills and test automation experience.
• Familiarity with LLM-based applications and prompt engineering.
• Knowledge of CI/CD pipelines and testing in production environments.

Responsibilities

• Develop and execute end-to-end automated test coverage across evaluations, APIs, UIs, and performance tests using Python.
• Implement testing strategies for GenAI features, including conversational AI and agentic workflows.
• Run and maintain automated test suites for prompt testing and regression tests.
• Support evaluation frameworks to measure GenAI performance on accuracy, relevance, safety, latency.
• Curate high-quality test datasets and golden examples for diverse scenarios and edge cases.
• Assist in adversarial testing and production alerting integration.
• Utilise internal tools to help developers validate GenAI code and assist with user acceptance testing.
• Document testing procedures and known issues clearly.

Skills

QA automation

Python coding

GenAI concepts

Test frameworks

API testing

CI/CD integration

Non-deterministic testing

Tools

Postman

REST Assured

Job description

Anaplan is pioneering a new QA automation role focused on GenAI/Agentic systems. As a QA Automation Engineer within AI, you'll design testing strategies, evaluation frameworks, and quality metrics for LLM-powered features to ensure reliability and trustworthiness.

You'll develop end-to-end automated test coverage across evaluations, APIs, UIs, and performance, implement GenAI test strategies, run regression tests, and help maintain datasets and golden examples for diverse scenarios and edge