QA / Automation Engineer Agentic AI

Compunnel, Inc.; Atlanta, Northern (GA, KY); Hybrid; USD 110,000 - 160,000; Full time

Job summary

Compunnel, Inc. seeks a QA / Automation Engineer to build and maintain automated testing and evaluation capabilities for an LLM-driven application. The role focuses on a Python-based framework for evaluating agentic systems, with LLM-as-judge evaluators and state validation.

You will work in a pod-based environment, sharing quality ownership and adapting as testing landscapes evolve, using Playwright where applicable while prioritizing Python-based AI evaluation.

Qualifications

• 7+ years of Python development for automation and evaluation.
• Experience testing AI/LLM-based systems and evaluators.
• Familiarity with Playwright for frontend automation.
• Comfort with collaborative, light QA processes.
• Ability to adapt to evolving testing landscapes.

Responsibilities

• Build and maintain automated tests for agentic, LLM-driven apps beyond UI test automation.
• Help develop a Python-based framework for evaluating agentic systems.
• Design and implement LLM-as-judge evaluators for AI behavior assessment.
• Develop state checks for conversational and agentic workflows.
• Collaborate with developers in distributed QA pods.
• Contribute to a light, UAT-style QA process with shared ownership.
• Evolve testing approaches as requirements develop.

Skills

Python

LLM testing

QA automation

Team collaboration

Tools

Playwright

Job description

We are seeking a QA / Automation Engineer to build and maintain automated testing and evaluation capabilities for an agentic, LLM-driven application. The primary focus of this role is developing a new Python-based framework for evaluating agentic systems, including LLM-as-judge evaluators, conversational and agentic state validation, and other non-standard evaluation approaches that extend beyond conventional UI automation. The engineer will work collaboratively with developers in a pod-based environment where quality ownership is shared and the testing landscape is continuously evolving.

Key Responsibilities

• Build and maintain automated tests for agentic, LLM-driven applications beyond conventional UI test automation.
• Help develop a new Python-based framework for evaluating agentic systems.
• Design and implement LLM-as-judge style evaluators for assessing AI-driven behavior and responses.
• Develop state checks and behavior validation for conversational and agentic workflows.
• Explore and implement non-standard evaluation methods that do not map directly to traditional scripted automation.
• Use Playwright for front-end automation where applicable while maintaining a primary focus on Python-based AI evaluation.
• Collaborate closely with developers within a distributed QA and development pod structure.
• Contribute to quality ownership within a light, UAT-style QA process where testing responsibilities are shared across the development team.
• Operate effectively within an evolving testing landscape and contribute to building the framework as requirements and evaluation approaches develop.

Required Qualifications

• 7+ years of experience with Python, with Python serving as the primary language for the evaluation and automation framework.
• Experience testing AI/LLM-based systems, including evaluators, LLM-as-judge techniques, or state and behavior validation for agentic or conversational systems.
• Strong aptitude for developing testing approaches for emerging and evolving AI-driven systems.
• Working knowledge of Playwright for front-end automation.
• Ability to work effectively within a collaborative development and QA environment where quality ownership is shared with developers.
• Comfort working with a light, UAT-style formal QA process rather than a heavily siloed QA model.
• Ability to operate effectively in an undefined and continuously evolving testing landscape.

Preferred Qualifications

• HCM industry domain knowledge.
• Experience building or evaluating agentic AI and conversational applications.