AI Automation Engineer

System One; Vienna (VA); Hybrid; USD 83,000 - 131,000; Full time

Benefits offered by this job

Health benefits

401(k) plan

Job summary

System One in Vienna, VA is seeking an AI Test Engineer to design and run automated tests for generative AI and AI-powered DevEx tools. You will build evaluation pipelines, validate model behavior, and ensure quality across CI/CD workflows.

The role emphasizes governance, reproducibility, and collaboration with cross-functional teams. The ideal candidate has strong Python skills, experience with test automation and AI evaluation techniques, and the ability to operate in a fast-paced, hybrid

Qualifications

• Bachelor's degree or equivalent in a technical field.
• 3+ years experience in software testing or AI/ML testing.
• Strong Python and scripting skills.
• Experience with automated test frameworks and CI/CD pipelines.
• Familiarity with AI models and evaluation techniques.

Responsibilities

• Help design, develop, and maintain AI testing and evaluation frameworks.
• Create and execute automated test suites for AI models and DevEx tools.
• Build automated evaluation pipelines integrated with CI/CD workflows.
• Perform functional, regression, and performance testing of AI-enabled apps.
• Analyze AI outputs to identify defects, bias, and reliability concerns.
• Maintain test datasets, benchmarks, and evaluation artifacts for AI systems.
• Collaborate with engineers to improve model quality and coverage.
• Define and report AI quality metrics like accuracy and latency.
• Support root-cause analysis and validation of AI defects.
• Document test strategies and recommendations for stakeholders.
• Contribute to governance and responsible AI efforts in testing.
• Support delivery of DevEx initiatives across the software lifecycle.

Skills

Python

CI/CD

Test Automation

Problem solving

Education

Bachelor's degree in Computer Science or related field

Tools

DeepEval

LangSmith

Azure AI

GitHub Copilot

Job description

Job Title: AI Test Engineer
Location: Vienna, VA
Type: Contract
Work Model: Hybrid – onsite and remote
Hours: 40.0

Responsibilities

• Help design, develop, and maintain AI testing and evaluation frameworks for generative AI, predictive AI, and AI-powered developer experience (DevEx) solutions.
• Create and execute automated test suites that validate AI model behavior, output quality, accuracy, reliability, and performance.
• Develop automated evaluation pipelines that integrate with CI/CD workflows and support continuous AI quality validation.
• Perform functional, regression, performance, and scenario‑based testing of AI‑enabled applications and services.
• Analyze AI-generated outputs to identify defects, inconsistencies, hallucinations, bias, performance issues, and other quality concerns.
• Build and maintain test datasets, benchmark scenarios, prompt libraries, and evaluation artifacts used to assess AI systems.
• Collaborate with software engineers, AI engineers, and platform teams to improve model quality and testing coverage.
• Define, measure, and report AI quality metrics such as accuracy, relevance, latency, consistency, robustness, and user satisfaction.
• Support root‑cause analysis, troubleshooting, and validation activities for AI‑related defects and incidents.
• Document test strategies, test results, evaluation methodologies, and recommendations for technical and non‑technical stakeholders.
• Contribute to AI governance and responsible AI efforts by supporting evaluation activities related to reliability, fairness, transparency, and safety.
• Support delivery of strategic DevEx initiatives and AI-enabled platform capabilities across the software development lifecycle.

Requirements

• Bachelor's degree in Computer Science, Information Systems, Software Engineering, Data Science, or a related technical field.
• 3+ years of experience in software testing, quality engineering, test automation, or AI/ML testing.
• Experience designing and executing automated test strategies for complex software applications.
• Strong programming and scripting skills in Python.
• Experience building automated test frameworks and test pipelines.
• Experience working with APIs, automated integration testing, and CI/CD environments.
• Understanding of machine learning, generative AI, large language models (LLMs), or AI-powered applications.
• Experience performing data validation, test result analysis, and defect investigation.
• Familiarity with modern testing methodologies, quality engineering practices, and software delivery lifecycles.
• Knowledge of statistical analysis and evaluation techniques used to validate system performance.
• Experience using source control systems and collaborative development practices.
• Strong analytical, problem‑solving, and troubleshooting skills.
• Excellent written and verbal communication skills.
• Ability to work independently in a fast‑paced, agile environment while collaborating effectively across teams.

Preferred Qualifications:

• Experience testing generative AI applications, LLM‑based solutions, or Retrieval‑Augmented Generation (RAG) systems.
• Experience with AI evaluation frameworks such as DeepEval, Ragas, LangSmith, or similar tools.
• Knowledge of LLM observability, monitoring, and traceability practices.
• Experience working with Azure AI, GitHub Copilot, OpenAI, Hugging Face, LangChain, or comparable AI platforms.
• Familiarity with prompt engineering, benchmark dataset creation, and AI output validation techniques.
• Understanding of Responsible AI principles and AI governance frameworks.
• Experience with performance testing, scalability testing, and reliability engineering.
• Experience supporting developer platforms, DevEx initiatives, or software engineering productivity tools.
• Experience working within enterprise or regulated technology environments.

Ideal Candidate Profile:

The ideal candidate is a hands‑on quality engineer who enjoys building testing capabilities rather than simply executing predefined test cases. They are curious about how AI systems behave, comfortable analyzing large volumes of AI‑generated outputs, and passionate about improving reliability through automation. They bring a strong software testing foundation, practical automation experience, and a desire to help create scalable AI quality processes from the ground up. They thrive in collaborative environments, communicate findings clearly, and can translate evaluation results into actionable improvements for engineering teams.

System One, and its subsidiaries including Mountain Ltd., are leaders in delivering outsourced services and workforce solutions across North America. We help clients get work done more efficiently and economically, without compromising quality. System One not only serves as a valued partner for our clients, but we offer eligible employees health and welfare benefits coverage options including medical, dental, vision, spending accounts, life insurance, voluntary plans, as well as participation in a 401(k) plan.

System One is an Equal Opportunity Employer. All qualified applicants will receive consideration for employment without regard to race, color, religion, sex (including pregnancy, childbirth, or related medical conditions), sexual orientation, gender identity, age, national origin, disability, family care or medical leave status, genetic information, veteran status, marital status, or any other characteristic protected by applicable federal, state, or local law.

#LI-EL1
#M2
Ref: #851-Rockville-S1