GenAI QA Automation Engineer | AI Testing & Reliability

Anaplan; Clayton (NC); On-site; USD 90,000 - 130,000; Full time

Job summary

Anaplan is expanding its AI QA focus with a role dedicated to GenAI/Agentic systems. The QA Automation Engineer will develop testing strategies and automate testing across evaluations, APIs, UIs, and performance tests in Python.

You will implement GenAI testing, including conversational AI and LLM-powered workflows, while maintaining regression suites and evaluation metrics for model behaviour and safety.

Qualifications

• Solid QA or test engineering experience, with exposure to AI/ML systems.
• Hands-on Python coding experience.
• Good understanding of GenAI technologies including LLMs, prompt engineering, and AI application patterns.
• Experience with API testing tools (Postman, REST Assured) and backend testing.
• Familiarity with CI/CD pipelines and automated testing integration.

Responsibilities

• Develop testing strategies and evaluation frameworks for GenAI features.
• Create end-to-end automated test coverage across APIs, UIs, and performance tests.
• Run and maintain automated test suites for prompt testing and regression checks.
• Support evaluation frameworks measuring accuracy, relevance, safety, and latency.
• Document testing procedures and known issues for the workstream.

Skills

QA Automation

Python

Pytest

API Testing

UI Testing

CI/CD

Education

Bachelor's degree in CS or related field

Tools

Postman

REST Assured

Job description

Anaplan is expanding its AI QA focus with a role dedicated to GenAI/Agentic systems. The QA Automation Engineer will develop testing strategies and automate testing across evaluations, APIs, UIs, and performance tests in Python.

You will implement GenAI testing, including conversational AI and LLM-powered workflows, while maintaining regression suites and evaluation metrics for model behaviour and safety.