GenAI QA Automation Engineer

Anaplan; Charlotte (NC); On-site; USD 110,000 - 160,000; Full time

Job summary

Anaplan in Charlotte, NC is seeking a QA Automation Engineer for GenAI/Agentic systems. You will design testing strategies and build automation to ensure GenAI features are reliable, accurate, and trustworthy.

You will develop end-to-end automated tests for evaluations, APIs, UIs, and performance, run regression suites, and contribute to evaluation datasets and golden examples.

Requires 8+ years of QA, Python, API testing, UI testing, and CI/CD experience.

Qualifications

• 8+ years of solid QA or test engineering experience in AI/ML contexts.
• Hands-on Python coding experience.
• Experience with GenAI, LLMs, and AI application patterns.
• Proficient in test automation frameworks and scripting.
• Knowledge of API, UI, and performance testing methodologies.
• Familiarity with CI/CD pipelines and testing integration.
• Ability to assess non-deterministic system criteria.

Responsibilities

• Develop and execute end-to-end automated test coverage across evaluations, APIs, UIs, and performance tests using Python.
• Implement testing strategies for GenAI features including conversational AI and agentic systems.
• Maintain automated test suites for prompts and regression tests.
• Support evaluation frameworks measuring GenAI accuracy, relevance, safety and latency.
• Contribute high-quality test datasets and golden examples.
• Assist with adversarial testing and integrate production alerting systems.
• Collaborate to validate GenAI code and support user acceptance testing.
• Document testing procedures and known issues clearly.

Skills

QA Automation

Python

GenAI testing

API testing

UI testing

CI/CD

Tools

Postman

REST Assured

Pytest

Selenium

Job description

Anaplan in Charlotte, NC is seeking a QA Automation Engineer for GenAI/Agentic systems. You will design testing strategies and build automation to ensure GenAI features are reliable, accurate, and trustworthy.

You will develop end-to-end automated tests for evaluations, APIs, UIs, and performance, run regression suites, and contribute to evaluation datasets and golden examples.

Requires 8+ years of QA, Python, API testing, UI testing, and CI/CD experience.