QA / Automation Engineer— Agentic AI

Advanced Tech Placement; Roseland (NJ); On-site; USD 90,000 - 130,000; Full time

Job summary

Advanced Tech Placement in the United States is seeking a QA / Automation Engineer to build and maintain automated tests for an agentic, LLM-driven application, and to create a new Python-based evaluation framework.

You will use Playwright for front-end automation where applicable, while shaping the Python-based evaluation layer for AI-driven behavior and collaborating in a pod structure with developers.

Qualifications

• Comfort with a light, UAT-style QA process and shared quality ownership.
• Ability to operate in an evolving testing landscape as the framework is built.
• Experience in designing Python-based evaluation frameworks for agentic systems.

Responsibilities

• Build and maintain automated tests for an agentic, LLM-driven application, extending beyond conventional UI test automation.
• Develop a new Python-based framework for evaluating agentic systems, including LLM-as-judge evaluators and state checks for conversational/agentic flows.
• Utilize Playwright for front-end automation where applicable, while focusing on the Python-based evaluation layer for AI-driven behavior.
• Collaborate closely with developers in a pod structure, as QA is distributed and collaborative rather than siloed.

Skills

Python

AI/LLM testing

Tools

Playwright

Job description

We are looking for a QA / Automation Engineer

This role involves building and maintaining automated tests for an agentic, LLM-driven application, focusing on creating a new Python-based framework for evaluating agentic systems.

Responsibilities

• Build and maintain automated tests for an agentic, LLM-driven application, extending beyond conventional UI test automation.
• Develop a new, Python-based framework for evaluating agentic systems, including LLM-as-judge style evaluators and state checks for conversational/agentic flows.
• Utilize Playwright for front-end automation where applicable, while focusing on the Python-based evaluation layer for AI-driven behavior.
• Collaborate closely with developers in a pod structure, as QA is distributed and collaborative rather than siloed.

Requirements

• Comfort operating with a light, UAT-style formal QA process, where quality ownership is shared with developers.
• Ability to work in an undefined, evolving testing landscape as the framework is being built and expanded.

Required Skills

• Proficiency in Python, as it is the primary language for the role's evaluation/automation framework.
• Experience or strong aptitude for testing AI/LLM-based systems, including evaluators, LLM-as-judge techniques, and state/behavior validation for agentic or conversational systems.
• Working knowledge of Playwright for the front-end automation layer, though it is not the core skill being hired for.

Preferred Skills

• Industry domain knowledge in HCM.