Senior GenAI QA Automation Engineer

Anaplan Inc; Greater London; Hybrid; GBP 90,000 - 120,000; Full time

Job summary

At Anaplan, we are pioneering a new role focused on quality assurance for GenAI/Agentic systems. As our Senior SDET for AI, you'll develop testing strategies, evaluation frameworks, and quality metrics tailored for LLM-powered applications.

You will design end-to-end automated test suites in Python across API/UI checks, performance tests, and prompt testing, while driving QA best practices and real-time monitoring to ensure reliable, trustworthy AI features in production.

Qualifications

• Deep understanding of GenAI technologies including LLMs and prompt engineering.
• Strong background in software testing methodologies across functional, integration and performance areas.
• Ability to design test cases and evaluation criteria for non-deterministic systems.
• Strong analytical and problem-solving skills with attention to detail.

Responsibilities

• Design and architect test plans, evaluation frameworks, and quality metrics for LLM-powered apps and agentic workflows.
• Develop robust, end-to-end automated test suites in Python for evaluations, API, UI, and performance.
• Create automated prompt testing systems and regression suites to detect model behavior changes.
• Establish evaluation criteria to measure GenAI quality across accuracy, relevance, safety, and latency.
• Perform adversarial testing to identify biases, hallucinations and vulnerabilities.
• Develop internal QA tools and testing frameworks for GenAI implementations.
• Deploy real-time monitoring and alerting to detect quality degradation in production.

Skills

GenAI technologies

LLMs

Prompt engineering

QA testing methodologies

API testing

CI/CD integration

Tools

Postman

REST Assured

Job description

At Anaplan, we are pioneering a new role focused on quality assurance for GenAI/Agentic systems. As our Senior SDET for AI, you'll develop testing strategies, evaluation frameworks, and quality metrics tailored for LLM-powered applications.

You will design end-to-end automated test suites in Python across API/UI checks, performance tests, and prompt testing, while driving QA best practices and real-time monitoring to ensure reliable, trustworthy AI features in production.