Senior GenAI QA Automation Engineer

Anaplan; Greater London; Vor Ort; GBP 85.000 - 120.000; Vollzeit

Zusammenfassung

Anaplan is hiring a Senior SDET for AI to focus on quality assurance for GenAI/Agentic systems. You will design testing strategies, evaluation frameworks, and quality metrics tailored for LLM-powered applications.

This role requires a mix of QA expertise, GenAI behavior understanding, and automation skills to ensure reliable AI features. You will develop end-to-end automated test suites in Python, create automated prompt testing, and establish evaluation criteria for accuracy, safety, latency.

Qualifikationen

• Deep understanding of GenAI technologies including LLMs, prompt engineering, and AI application patterns.
• Strong hands-on Python coding experience.
• Expert experience with test automation frameworks and scripting (Python, JavaScript, Selenium, Pytest).
• Strong background of software testing methodologies (functional, integration, regression, performance, security testing).
• Proven ability to design test cases and evaluation criteria for non-deterministic systems.

Aufgaben

• Design and architect comprehensive test plans, evaluation frameworks, and quality metrics designed specifically for LLM-powered applications and agentic workflows.
• Develop robust, end-to-end automated test suites in Python across evaluations, API checks, UI checks, and performance tests.
• Create automated prompt testing systems and regression suites to proactively detect unintended changes in model behaviour.
• Establish multi-dimensional evaluation criteria to measure and track GenAI quality across accuracy, relevance, safety, consistency, and latency.
• Perform adversarial testing (red-teaming) to identify and mitigate model hallucinations, biases, or security vulnerabilities.
• Develop internal tools and testing frameworks that enable software engineering teams to seamlessly validate their own GenAI implementations.
• Deploy real-time monitoring and alerting systems to quickly detect and flag quality degradation of AI features in production.
• Own the overall quality and testing strategy for your workstream, maintaining high-fidelity test datasets ("golden examples") and defining QA best practices.

Kenntnisse

GenAI

LLMs

Prompt engineering

Python

JavaScript

Selenium

Pytest

Test automation

API testing

CI/CD

QA strategies

Regression testing

Tools

Postman

REST Assured

Selenium

Jobbeschreibung

Anaplan is hiring a Senior SDET for AI to focus on quality assurance for GenAI/Agentic systems. You will design testing strategies, evaluation frameworks, and quality metrics tailored for LLM-powered applications.

This role requires a mix of QA expertise, GenAI behavior understanding, and automation skills to ensure reliable AI features. You will develop end-to-end automated test suites in Python, create automated prompt testing, and establish evaluation criteria for accuracy, safety, latency.