AI QA Engineer for LLM & GenAI Testing & Automation

Compunnel, Inc.; New York, Northern (NY, KY); Hybrid; USD 140,000 - 190,000; Full time

Job summary

Compunnel, Inc. seeks an Evaluation Engineer - AI Models in New York to evaluate and validate AI/ML and Generative AI models across business and technical use cases.

This role combines extensive Quality Engineering and software testing experience with hands-on AI model evaluation, production-grade LLM testing, test automation, and data-driven analysis. Strong Selenium and Playwright expertise is required, along with excellent communication skills and the ability to collaborate with engineering,

Qualifications

• 10+ years of experience in Quality Assurance, Quality Engineering, Software Testing, or related disciplines.
• 2+ years of hands-on experience in AI model evaluation, Generative AI testing, or ML validation.
• Strong hands-on experience with Selenium and Playwright.
• Strong understanding of AI/ML concepts, LLM behavior, prompt evaluation, and model testing methodologies.
• Experience testing production-grade LLMs and AI-powered applications.
• Experience with API testing, test automation frameworks, and data validation techniques.
• Understanding of evaluation metrics such as precision, recall, accuracy, grounding, relevance, and hallucination detection.
• Experience creating automated and manual test strategies and frameworks.
• Strong analytical and problem-solving skills.
• Excellent verbal and written communication skills.
• Ability to work independently and collaborate effectively with cross-functional teams.

Responsibilities

• Evaluate and validate AI models across business and technical use cases.
• Design evaluation strategies, validate model accuracy and performance, detect hallucinations and other quality issues.
• Develop automated and manual testing frameworks and data-driven analysis.

Skills

Quality Assurance

Quality Engineering

AI model evaluation

Test automation

Data validation

Analytical thinking

Communication skills

Cross-functional collaboration

Tools

Selenium

Playwright

Job description

Compunnel, Inc. seeks an Evaluation Engineer - AI Models in New York to evaluate and validate AI/ML and Generative AI models across business and technical use cases.

This role combines extensive Quality Engineering and software testing experience with hands-on AI model evaluation, production-grade LLM testing, test automation, and data-driven analysis. Strong Selenium and Playwright expertise is required, along with excellent communication skills and the ability to collaborate with engineering,