AI Test Engineer

ECS; Fairfax (VA); On-site; USD 125,000 - 150,000; Full time

Job summary

ECS Federal LLC is seeking an AI Testing Engineer in Virginia to support CBP''s Project Outrider. The role focuses on building and executing test plans for AI models, identifying defects, and collaborating with engineering teams to enhance model quality.

The position requires 3+ days on-site at Fairfax or Ashburn, with responsibilities spanning automation, dataset checks, API testing, and CI/CD integration to ensure reliable AI-driven solutions.

Qualifications

• U.S. Citizenship; Must be able to obtain and maintain a Public Trust determination.
• Bachelor’s degree in Computer Science, Electrical Engineering, or a related technical field.
• 2-4+ years of AI Test and Validation experience.
• Experience working on custom development software projects.
• Strong collaboration with cross-functional teams to identify testing requirements and optimize test coverage.
• Experience with system testing metrics, best practices, and methodologies.
• Utilization of Python and Selenium for automated testing of machine learning models.
• Experience working with software development teams.
• Experience analyzing test results and providing actionable insights to improve model accuracy.
• Experience designing and implementing comprehensive test strategies for AI-driven applications.
• Experience participating in requirements and design reviews, developing test plans, writing and executing test cases in a fast-paced software development environment.
• Experience conducting data analysis, data verification and problem-solving.
• Experience with statistical reporting and analysis.
• Competency in developing and executing test plans for various AI applications, ensuring compliance with industry standards.
• Expertise in one or more scripting languages (e.g., Unix/Linux, PERL, Java script) and relational database technologies (SQL, schema design).
• Ability to work as a member of a self-organizing team without daily direct supervision.
• Ability to communicate effectively within a large, dynamic team environment.
• Ability to support three days in-office.
• Willingness to try new things and grow.

Responsibilities

• Develop and execute test plans for AI models and applications, identifying defects and anomalies in outputs.
• Collaborate with development teams to improve AI model performance and ensure quality standards.
• Implement automation frameworks for AI evaluation, including dataset checks, API testing, and CI/CD integration.
• Define AI quality metrics and KPIs such as accuracy, latency, cost, and fairness.

Skills

Python

Selenium

AI testing

CI/CD

Data analysis

SQL

Unix/Linux scripting

JavaScript

Team collaboration

Education

Bachelor's degree in Computer Science, Electrical Engineering, or related field

Master's degree (preferred)

Tools

Unix/Linux

SQL

Job description

Position Overview

The AI Testing Engineer will be part of a team charged with providing support in the areas of model analysis and model development capabilities in response to Custom and Border Protection’s (CBP) Project Outrider. The position is contingent upon contract award and requires a minimum of 3 business days onsite at our Fairfax, VA corporate office and/or our Ashburn, VA customer site.

Responsibilities

• The development and execution of test plans for AI models and applications, identifying, documenting, and analyzing defects, inconsistencies, and anomalies in AI outputs.
• Collaboration with development teams to improve AI model performance and ensure quality standards.
• Implementing automation frameworks for AI evaluation, including dataset checks, API testing, and CI/CD integration.
• Defining AI quality metrics and KPIs such as accuracy, latency, cost, and fairness.

Compensation

Salary Range: 125,000-150,000

Required Skills

• U.S. Citizenship; Must be able to obtain and maintain a Public Trust determination.
• Bachelor’s degree in Computer Science, Electrical Engineering, or a related technical field.
• 2-4+ years of AI Test and Validation experience.
• Experience working on custom development software projects.
• Strong collaboration with cross-functional teams to identify testing requirements and optimize test coverage.
• Experience with system testing metrics, best practices, and methodologies.
• Utilization of Python and Selenium for automated testing of machine learning models.
• Experience working with software development teams.
• Experience analyzing test results and providing actionable insights to improve model accuracy.
• Experience designing and implementing comprehensive test strategies for AI-driven applications.
• Experience participating in requirements and design reviews, developing test plans, writing and executing test cases in a fast-paced software development environment.
• Experience conducting data analysis, data verification and problem-solving.
• Experience with statistical reporting and analysis.
• Competency in developing and executing test plans for various AI applications, ensuring compliance with industry standards.
• Expertise in one or more scripting languages (e.g., Unix/Linux, PERL, Java script) and relational database technologies (SQL, schema design).
• Ability to work as a member of a self-organizing team without daily direct supervision.
• Ability to communicate effectively within a large, dynamic team environment.
• Ability to support three days in-office.
• Willingness to try new things and grow.

Desired Skills

• Master’s degree in Computer Science, Electrical Engineering, or a related technical field.
• Experience working with DHS customers.
• Prior experience supporting DoW, Intelligence Community, or other U.S. Government programs.
• Experience implementing regression testing protocols to catch errors in system updates.
• Experience documenting testing processes and outcomes for stakeholder review.

EEO Statement

ECS Federal LLC is an equal opportunity employer and does not discriminate or allow discrimination on the basis of any characteristic protected by law. All qualified applicants will receive consideration for employment without regard to disability, status as a protected veteran or any other status protected by applicable federal, state, or local jurisdiction law.