GenAI QA Automation Engineer

Anaplan; United States; On-site; USD 105,000 - 140,000; Full time

Job summary

Anaplan is hiring a QA Automation Engineer focused on GenAI and agentic systems. You’ll design testing strategies, evaluation frameworks, and quality metrics to ensure our LLM-powered features are reliable, accurate, and trustworthy across evaluations, APIs, UIs, and performance tests using Python.

The role requires hands-on Python, experience with test automation frameworks, API testing tools, and familiarity with CI/CD.

Qualifications

• Solid QA or test engineering experience with exposure to AI/ML systems.
• Hands-on Python coding experience.
• Experience with test automation frameworks and scripting (Python, Pytest, API, and UI Testing).
• Knowledge of GenAI technologies including LLMs, prompts, and AI application patterns.
• Experience with API testing tools like Postman and REST Assured.
• Familiarity with CI/CD pipelines and automated testing integration.
• Ability to test non-deterministic systems.

Responsibilities

• Develop and execute end-to-end automated test coverage across evaluations, APIs, UIs, and performance tests using Python.
• Implement testing strategies for GenAI features, including conversational AI, agentic systems, and LLM-powered workflows.
• Run and maintain automated test suites for prompt testing, including regression tests for model behaviour.
• Support and maintain evaluation frameworks to measure GenAI performance across accuracy, relevance, safety, consistency, and latency.
• Curate high-quality test datasets and golden examples representing diverse user scenarios and edge cases.
• Assist with adversarial testing to identify potential model hallucinations or biases; support production alerting integration.
• Utilize internal tools to help developers validate GenAI code and assist with user acceptance testing.
• Document testing procedures and known issues in clear, accessible formats.

Skills

QA Automation

Python Coding

GenAI Knowledge

Test Frameworks

API Testing

CI/CD

Tools

Postman

REST Assured

Pytest

Job description

Anaplan is hiring a QA Automation Engineer focused on GenAI and agentic systems. You’ll design testing strategies, evaluation frameworks, and quality metrics to ensure our LLM-powered features are reliable, accurate, and trustworthy across evaluations, APIs, UIs, and performance tests using Python.

The role requires hands-on Python, experience with test automation frameworks, API testing tools, and familiarity with CI/CD.