AI QA Engineer

Cavendish Professionals; Town of Italy (NY); On-site; USD 95,000 - 120,000; Full time

Job summary

A leading recruitment agency is seeking an AI QA Engineer to define and lead the QA strategy for AI and data-driven systems. The role requires 3-6 years of experience in QA or SDET, proficiency in Python scripting, and a strong understanding of ML evaluation tools. Responsibilities include creating test plans, conducting scenario-based tests, and integrating tests into CI/CD pipelines. The position is located in New York and promotes equal opportunity employment.

Qualifications

• 3–6+ years of experience in QA or SDET, ideally with AI/ML testing.
• Capability to evaluate data correctness and retrieval accuracy.
• Experience in designing test plans for non-functional and behavioral requirements.

Responsibilities

• Define and own the end-to-end QA strategy across AI components.
• Conduct scenario-based tests and regression tests.
• Integrate automated tests into CI/CD pipelines.

Skills

Python scripting skills

ML evaluation tools

Experience with LLM/RAG testing

Knowledge of vector databases

Understanding of CI/CD pipelines

Attention to edge cases

Excellent communication skills

Job description

Our client is seeking an AI QA Engineer to ensure reliable, safe, and high-quality AI and data-driven systems in fast-paced, experimental projects.

Roles and Responsibilities

• Define and own the end-to-end QA strategy across UI, backend, data, and AI components.
• Design and implement test plans covering functional, non-functional, and behavioral requirements.
• Build automated and manual evaluation suites for LLMs, retrieval systems, and agent workflows.
• Conduct scenario‑based tests, regression tests, and red‑team exercises to uncover edge cases and risks.
• Validate data quality, embedding correctness, retrieval accuracy, and monitor for model drift or hallucinations.
• Define metrics, logs, dashboards, and alerts to monitor AI behavior, latency, cost, and errors.
• Detect and elevate reliability, bias, and performance issues early in the delivery cycle.
• Manage defect triage workflows, categorize failures across UI, API, data, and model layers, and ensure clear reproduction documentation.
• Integrate automated tests into CI/CD pipelines to catch regressions early.
• Support pods in delivering safe, reliable, and stable AI behavior for all experimental and production features.

Requirements

• 3–6+ years of experience in QA, SDET, or evaluation-focused ML/AI testing, preferably with nondeterministic or probabilistic systems.
• Strong Python scripting skills for test automation, scenario evaluation, and defect detection.
• Experience with ML evaluation tools, LLM/RAG testing, or model benchmarking suites.
• Familiarity with vector databases, retrieval systems, multi‑agent workflows, and AI pipelines.
• Understanding of CI/CD pipelines, DevOps tooling, and observability platforms.
• Ability to validate data correctness, embeddings, retrieval accuracy, and ranking/precision metrics.
• Strong instincts for edge cases, risk modes, adversarial failures, and failure‑mode analysis.
• Curious, systematic, and detail‑oriented mindset, with high ownership and discipline.
• Excellent communication skills for clear, actionable defect reporting.

For more information – please apply for this job or send your CV directly and we will contact you to provide further details.

Cavendish (Recruitment) Professionals Ltd are proud to be an equal opportunity employer and we believe that inclusivity begins with the candidate experience. All qualified applicants will receive consideration for employment regardless of gender, race, age, sexual orientation, religion, or belief.