AI Agentic QA Engineer using Python & Playwright

Advanced Tech Placement; Roseland (NJ); On-site; USD 90,000 - 130,000; Full time

Job summary

Advanced Tech Placement in the United States is seeking a QA / Automation Engineer to build and maintain automated tests for an agentic, LLM-driven application, and to create a new Python-based evaluation framework.

You will use Playwright for front-end automation where applicable, while shaping the Python-based evaluation layer for AI-driven behavior and collaborating in a pod structure with developers.

Qualifications

• Comfort with a light, UAT-style QA process and shared quality ownership.
• Ability to operate in an evolving testing landscape as the framework is built.
• Experience in designing Python-based evaluation frameworks for agentic systems.

Responsibilities

• Build and maintain automated tests for an agentic, LLM-driven application, extending beyond conventional UI test automation.
• Develop a new Python-based framework for evaluating agentic systems, including LLM-as-judge evaluators and state checks for conversational/agentic flows.
• Utilize Playwright for front-end automation where applicable, while focusing on the Python-based evaluation layer for AI-driven behavior.
• Collaborate closely with developers in a pod structure, as QA is distributed and collaborative rather than siloed.

Skills

Python

AI/LLM testing

Tools

Playwright

Job description

Advanced Tech Placement in the United States is seeking a QA / Automation Engineer to build and maintain automated tests for an agentic, LLM-driven application, and to create a new Python-based evaluation framework.

You will use Playwright for front-end automation where applicable, while shaping the Python-based evaluation layer for AI-driven behavior and collaborating in a pod structure with developers.