Senior AI Test Engineer for GenAI & LLM Apps

Anaplan; North Carolina; Hybrid; USD 120,000 - 180,000; Full time

Job summary

Anaplan, a leader in AI-infused scenario planning, seeks a Senior Software Development Engineer in Test to ensure quality across our GenAI-powered platform. You will design test plans, build end-to-end automated tests, and craft evaluation metrics for AI-driven workflows.

You will champion testing discipline, explore non-deterministic system behavior, and work with CI/CD to deliver reliable AI features, with attention to accuracy, safety, and latency.

Qualifications

• Design test plans and evaluation criteria for GenAI-powered features.
• Apply functional, integration, regression, performance tests and security testing.
• Create test cases for non-deterministic AI systems.
• Work with CI/CD to integrate automated tests.

Responsibilities

• Design and architect comprehensive test plans, evaluation frameworks, and quality metrics for LLM-powered applications.
• Develop end-to-end automated test suites in Python across evaluations, API checks, UI checks, and performance tests.
• Create automated prompt testing systems and regression suites to detect changes in model behavior.
• Establish evaluation criteria for accuracy, relevance, safety, and latency of GenAI features.
• Perform adversarial testing to identify hallucinations and biases in AI models.
• Develop internal testing tools to support GenAI deployments.
• Deploy real-time monitoring to flag quality degradation in production.

Skills

GenAI technologies (LLMs)

Testing methodologies

Non-deterministic test design

Analytical reasoning

CI/CD & automation testing

Tools

Postman

REST Assured

Job description

Anaplan, a leader in AI-infused scenario planning, seeks a Senior Software Development Engineer in Test to ensure quality across our GenAI-powered platform. You will design test plans, build end-to-end automated tests, and craft evaluation metrics for AI-driven workflows.

You will champion testing discipline, explore non-deterministic system behavior, and work with CI/CD to deliver reliable AI features, with attention to accuracy, safety, and latency.