AI Quality & Test Engineer

simpletechnologysolutions; Washington (District of Columbia); Remote; USD 140,000 - 180,000; Full time

Benefits offered by this job

HUBZone incentives

Job summary

Simple Technology Solutions is seeking a Senior QA/Test Automation Engineer to lead testing for AI-enabled government solutions. The role emphasizes building automated end-to-end tests, AI evaluation harnesses, and integrating tests into CI/CD pipelines.

Remote work with occasional travel, preference for DC Metro region and HUBZone incentives. The ideal candidate will have 8+ years in software quality engineering, strong Python/TypeScript automation skills, and familiarity with LLM evaluation

Qualifications

• Bachelor’s degree or equivalent professional experience.
• 8+ years of software quality engineering or test automation
• Experience testing applications built on large language models or other AI/ML components
• Experience leading user-acceptance testing with business or mission users, ideally in federal or regulated environment
• Strong test automation skills in Python or TypeScript with Playwright, pytest, or Jest
• Practical knowledge of LLM evaluation techniques, including golden datasets and rubric-based evaluation

Responsibilities

• Define the test strategy for AI applications across DEV, TST, STG, and PRD.
• Design, build, and execute end-to-end and user-acceptance tests.
• Build evaluation harnesses for AI behavior and safety checks.
• Automate tests in the CI/CD pipeline and maintain quality gates.
• Triage, reproduce, and track defects in Jira and coordinate with engineers.

Skills

Python

Test automation

Playwright

CI/CD

LLM testing

Education

Bachelor's degree in CS or related field

Equivalent professional experience

Tools

Jira

Playwright

pytest

Jest

Bedrock

Job description

At Simple Technology Solutions, our people are our priority. We know our team members are more than employees-they're parents, friends, volunteers, artists, and athletes. That's why we offer flexibility to help them thrive personally and professionally while delivering exceptional solutions to our Federal Government clients.

Our culture is built on collaboration, continuous learning, and excellence. We are mentors and thought leaders who share knowledge and foster growth. Recognized as a "Best Place to Work," we believe a range of perspectives helps us drive innovation and exceed customer expectations. At STS, taking care of our people isn't a perk-it's the standard.

As a HUBZone company, we also offer special incentives for team members living in qualified HUBZones. Check out the HUBZone map HERE to see if you qualify!

Quick Position Overview:

• US Citizenship and the ability to obtain and maintain a Public Trust clearance are required
• Bachelor's Degree is required
• Minimum of 8 years' position related experience is required
• AWS Certified AI Practitioner or Developer (Associate) certification preferred for hire; the AWS and Anthropic Claude certifications listed under Required Certifications must be earned within 12 months of hire
• Remote position with occasional travel to client sites and STS events
• Candidates living in the DC Metro region (within driving distance for quarterly in-person team events) are strongly preferred; STS is a HUBZone company, so residence in a HUBZone is favored for remote candidates

The Role:

• Define the test strategy for each AI application project from the requirements trio, including unit, integration, end-to-end, user-acceptance, security, and AI-behavior testing, and the coverage and gate thresholds the project will use.
• Design, build, and execute end-to-end and user-acceptance tests that exercise complete use cases as real users would, across DEV, TST, STG, and PRD.
• Build evaluation harnesses for AI behavior, including golden datasets, rubric-based and model-assisted grading, regression suites for prompts and retrieval, and guardrail and safety checks.
• Operate as the tester in the team's AI development workflow, using and improving the shared testing agents and skills, and ensuring tests derived from use cases exist before application code is written.
• Automate tests in the CI/CD pipeline in partnership with the platform engineer, and own the quality gates that govern promotion between environments.
• Validate acceptance criteria with the business analyst and real users, facilitate user-acceptance testing sessions, and capture results as release evidence.
• Triage, reproduce, and track defects in Jira, drive them to resolution with engineers, and analyze defect trends to improve upstream requirements and development practices.
• Test on AWS-native AI stacks, including Amazon Bedrock models, agents, and knowledge bases, when a client requires them in place of STS's standard toolchain.
• Assess performance, cost, and resilience of AI features under realistic load and failure conditions, and report findings with recommendations.
• Contribute test standards, templates, and tooling back to the STS SDLC, and coach the team on test-driven development and AI evaluation practices.

Education and Experience:

• Required: Bachelor's degree in Computer Science, Software Engineering, Information Systems, or a related technical field, or equivalent professional experience in addition to the minimum below
• Minimum of 8 years of software quality engineering or test automation experience, including at least 2 years testing applications built on large language models or other AI/ML components
• Demonstrated experience building automated end-to-end test suites integrated into CI/CD pipelines and defining quality gates for a delivery team
• Experience leading user-acceptance testing with business or mission users, ideally in a federal or regulated environment
• Strong test automation skills in Python or TypeScript using frameworks such as Playwright, pytest, or Jest, plus API testing tools
• Practical knowledge of LLM evaluation techniques, including golden datasets, rubric and model-graded evaluation, retrieval quality metrics, and prompt and model regression testing
• Working