Agentic AI QA Engineer: Python Automation & Evaluation

Compunnel, Inc.; Atlanta, Northern (GA, KY); Hybrid; USD 110,000 - 160,000; Full time

Job summary

Compunnel, Inc. seeks a QA / Automation Engineer to build and maintain automated testing and evaluation capabilities for an LLM-driven application. The role focuses on a Python-based framework for evaluating agentic systems, with LLM-as-judge evaluators and state validation.

You will work in a pod-based environment, sharing quality ownership and adapting as testing landscapes evolve, using Playwright where applicable while prioritizing Python-based AI evaluation.

Qualifications

• 7+ years of Python development for automation and evaluation.
• Experience testing AI/LLM-based systems and evaluators.
• Familiarity with Playwright for frontend automation.
• Comfort with collaborative, light QA processes.
• Ability to adapt to evolving testing landscapes.

Responsibilities

• Build and maintain automated tests for agentic, LLM-driven apps beyond UI test automation.
• Help develop a Python-based framework for evaluating agentic systems.
• Design and implement LLM-as-judge evaluators for AI behavior assessment.
• Develop state checks for conversational and agentic workflows.
• Collaborate with developers in distributed QA pods.
• Contribute to a light, UAT-style QA process with shared ownership.
• Evolve testing approaches as requirements develop.

Skills

Python

LLM testing

QA automation

Team collaboration

Tools

Playwright

Job description

Compunnel, Inc. seeks a QA / Automation Engineer to build and maintain automated testing and evaluation capabilities for an LLM-driven application. The role focuses on a Python-based framework for evaluating agentic systems, with LLM-as-judge evaluators and state validation.

You will work in a pod-based environment, sharing quality ownership and adapting as testing landscapes evolve, using Playwright where applicable while prioritizing Python-based AI evaluation.