AI Test Engineer — Automate AI QA & Evaluation

Apex Systems; Merrifield (VA); Hybrid; USD 100,000 - 140,000; Full time

Benefits offered by this job

Medical, dental, vision

401K with company match

ESPP

HSA

EAP with counseling

Certification prep and training

Job summary

Everforth Apex Systems seeks an AI Test Engineer to build and scale automated testing for AI-enabled products. You will work with AI engineers, platform teams, and product stakeholders to validate model behavior and performance in a hybrid environment across multiple US sites.

The role emphasizes hands-on testing, data validation, and developing evaluation pipelines that integrate with existing CI/CD workflows. A background in ML/AI testing and strong Python skills are required.

Qualifications

• Bachelor's degree in Computer Science, Information Systems, Software Engineering, Data Science, or a related field.
• 3+ years of experience in software testing, quality engineering, test automation, or AI/ML testing.
• Strong Python and automation framework skills; experience with APIs, CI/CD and testing strategies.

Responsibilities

• Design, develop, and maintain AI testing and evaluation frameworks for generative and predictive AI.
• Create and execute automated test suites validating model behavior, quality, and performance.
• Develop automated evaluation pipelines integrated with CI/CD workflows.
• Perform functional, regression, performance, and scenario-based testing of AI-enabled apps.
• Analyze AI outputs to identify defects, bias, hallucinations, and latency issues.
• Build and maintain test datasets, benchmarks, and evaluation artifacts.

Skills

Python

Test automation

CI/CD

APIs

LLMs

ML understanding

Data validation

Education

Bachelor's degree in CS/related field

Tools

Azure AI

GitHub Copilot

OpenAI

Hugging Face

LangChain

Job description

Everforth Apex Systems seeks an AI Test Engineer to build and scale automated testing for AI-enabled products. You will work with AI engineers, platform teams, and product stakeholders to validate model behavior and performance in a hybrid environment across multiple US sites.

The role emphasizes hands-on testing, data validation, and developing evaluation pipelines that integrate with existing CI/CD workflows. A background in ML/AI testing and strong Python skills are required.