Lead AI QA Engineer for LLM & AI Testing

Jobtailor; Hoboken (NJ); On-site; USD 120,000 - 160,000; Full time

Job summary

Jobtailor is seeking a Senior Quality Engineer to lead testing across AI-driven features and non-deterministic systems in a fast-paced product environment. You will define quality strategies, own test plans, and drive automation across UI, API, and backend layers.

The role requires hands-on experience with Playwright and JavaScript/TypeScript or Java, plus a track record in Agile delivery and end-to-end QE leadership for complex projects.

Qualifications

• 8+ years of experience in Quality Engineering or Quality Assurance.

Responsibilities

• Define quality strategies and own test planning from requirements through release, partnering closely with cross-functional teams.
• Design and execute scalable functional, regression, and integration testing strategies.
• Apply hands-on automation framework expertise across UI, API, and backend layers to support major product initiatives.
• Test AI-driven functionality, including LLM agents, multi-agent workflows, and RAG pipelines.
• Validate AI outputs for accuracy, relevance, consistency, and edge cases across non-deterministic systems.
• Own evaluation frameworks (e.g., RAGAS), test datasets, and regression baselines for AI features.
• Stay current with the evolving landscape of LLM evaluation, agent testing, and AI observability, and translate emerging techniques into practical testing approaches.
• Lead QE team for your project by setting technical direction, defining quality strategy, and ensuring delivery confidence.
• Contribute to QE standards and frameworks within your project scope, establishing practices that other engineers can adopt.
• Integrate automated tests into CI/CD pipelines and partner with engineering leads to embed quality into the delivery process.
• Leverage data, risk signals, and production insights to continuously improve test coverage and effectiveness.

Skills

Quality engineering

AI/LLM testing

Automation testing

Playwright

JavaScript/TypeScript

Java

API testing

Agile

Tools

Playwright

JavaScript/TypeScript

Java

Job description

Jobtailor is seeking a Senior Quality Engineer to lead testing across AI-driven features and non-deterministic systems in a fast-paced product environment. You will define quality strategies, own test plans, and drive automation across UI, API, and backend layers.

The role requires hands-on experience with Playwright and JavaScript/TypeScript or Java, plus a track record in Agile delivery and end-to-end QE leadership for complex projects.