Automation QA w/ AI testing

Compunnel Inc.; New York (NY); On-site; USD 140,000 - 190,000; Full time

Job summary

Compunnel, Inc. seeks an Evaluation Engineer - AI Models in New York to evaluate and validate AI/ML and Generative AI models across business and technical use cases.

This role combines extensive Quality Engineering and software testing experience with hands-on AI model evaluation, production-grade LLM testing, test automation, and data-driven analysis. Strong Selenium and Playwright expertise is required, along with excellent communication skills and the ability to collaborate with engineering,

Qualifications

• 10+ years of experience in Quality Assurance, Quality Engineering, Software Testing, or related disciplines.
• 2+ years of hands-on experience in AI model evaluation, Generative AI testing, or ML validation.
• Strong hands-on experience with Selenium and Playwright.
• Strong understanding of AI/ML concepts, LLM behavior, prompt evaluation, and model testing methodologies.
• Experience testing production-grade LLMs and AI-powered applications.
• Experience with API testing, test automation frameworks, and data validation techniques.
• Understanding of evaluation metrics such as precision, recall, accuracy, grounding, relevance, and hallucination detection.
• Experience creating automated and manual test strategies and frameworks.
• Strong analytical and problem-solving skills.
• Excellent verbal and written communication skills.
• Ability to work independently and collaborate effectively with cross-functional teams.

Responsibilities

• Evaluate and validate AI models across business and technical use cases.
• Design evaluation strategies, validate model accuracy and performance, detect hallucinations and other quality issues.
• Develop automated and manual testing frameworks and data-driven analysis.

Skills

Quality Assurance

Quality Engineering

AI model evaluation

Test automation

Data validation

Analytical thinking

Communication skills

Cross-functional collaboration

Tools

Selenium

Playwright

Job description

The Evaluation Engineer - AI Models will be responsible for evaluating and validating AI/ML and Generative AI models across business and technical use cases. This role combines extensive Quality Engineering and software testing experience with hands-on AI model evaluation, production-grade LLM testing, test automation, and data-driven analysis. The engineer will design evaluation strategies, validate model accuracy and performance, detect hallucinations and other quality issues, and develop automated and manual testing frameworks. Strong Selenium and Playwright expertise is required, along with excellent communication skills and the ability to collaborate with engineering, product, data science, and business stakeholders. Financial Services or Wealth Management domain experience is preferred.

Required Qualifications

• 10+ years of experience in Quality Assurance, Quality Engineering, Software Testing, or related disciplines.
• 2+ years of hands-on experience in AI model evaluation, Generative AI testing, or ML validation.
• Strong hands-on experience with Selenium and Playwright.
• Strong understanding of AI/ML concepts, LLM behavior, prompt evaluation, and model testing methodologies.
• Experience testing production-grade LLMs and AI-powered applications.
• Experience with API testing, test automation frameworks, and data validation techniques.
• Understanding of evaluation metrics such as precision, recall, accuracy, grounding, relevance, and hallucination detection.
• Experience creating automated and manual test strategies and frameworks.
• Strong analytical and problem-solving skills.
• Excellent verbal and written communication skills.
• Ability to work independently and collaborate effectively with cross-functional teams.