GenAI QA Automation Engineer

Anaplan; North Carolina; Hybrid; USD 110,000 - 160,000; Full time

Benefits offered by this job

Competitive salary

Health benefits

Job summary

Anaplan in the United States (North Carolina) is seeking a QA Automation Engineer focused on GenAI/Agentic systems. You will develop testing strategies, evaluation frameworks, and quality metrics for LLM-powered applications, blending QA expertise with automation to ensure reliable AI features.

The role requires hands-on Python, experience with Pytest and API/UI testing, and strong knowledge of GenAI technologies, pipelines, and CI/CD integrations.

Qualifications

• 8+ years of solid QA or test engineering experience with exposure to AI/ML systems.
• Hands-on Python coding experience.
• Strong understanding of GenAI technologies including LLMs and prompt engineering.
• Solid experience with test automation frameworks and scripting (Python, Pytest, API and UI Testing).
• Practical knowledge of software testing methodologies (functional, integration, regression, performance, security testing).
• Ability to execute test cases and evaluate criteria for non-deterministic systems.
• Experience with API testing tools (Postman, REST Assured) and backend testing.
• Familiarity with CI/CD pipelines and automated testing integration.

Responsibilities

• Develop and execute end-to-end automated test coverage across evaluations, APIs, UIs, and performance tests using Python.
• Implement testing strategies for GenAI features, including conversational AI, agentic systems, and LLM-powered workflows within your workstream.
• Run and maintain automated test suites for prompt testing, including regression tests designed to detect unintended changes in model behaviour.
• Support and maintain evaluation frameworks to measure GenAI performance across accuracy, relevance, safety, consistency, and latency.
• Contribute to curating high-quality test datasets and "golden examples" that represent diverse user scenarios and edge cases.
• Assist with adversarial testing to identify potential model hallucinations or biases, and support integrating production alerting systems.
• Utilize internal tools and testing frameworks to help developers easily validate their GenAI code, and assist with user acceptance testing.
• Take accountability for your workstream's automated coverage , documenting testing procedures and known issues in clear, accessible formats.

Skills

QA engineering

Python

GenAI concepts

LLMs

Prompt engineering

Test automation

APIs testing

UI testing

CI/CD

Non-deterministic tests

Tools

Postman

REST Assured

Pytest

UI testing frameworks

CI/CD tools

Job description

Anaplan in the United States (North Carolina) is seeking a QA Automation Engineer focused on GenAI/Agentic systems. You will develop testing strategies, evaluation frameworks, and quality metrics for LLM-powered applications, blending QA expertise with automation to ensure reliable AI features.

The role requires hands-on Python, experience with Pytest and API/UI testing, and strong knowledge of GenAI technologies, pipelines, and CI/CD integrations.