Senior AI Quality & Test Engineer (Remote)

Simple Technology Solutions, LLC; Washington (District of Columbia); Remote; USD 120,000 - 180,000; Full time

Benefits offered by this job

Competitive compensation

Health insurance

401(k) retirement savings plan

Work from home stipend

Unlimited PTO

Paid holidays

Professional development budget

Job summary

Simple Technology Solutions, LLC is seeking an AI Quality & Test Engineer to own the end-to-end testing of AI applications built on AWS-native services. You will design test strategies, execute UAT and integration tests, and ensure AI outputs meet acceptance criteria in a remote, nationally distributed team.

You will lead automated testing in CI/CD, collaborate with business analysts and users, and help advance QA standards across the STS SDLC, including security and regulatory compliance for

Qualifications

• Bachelor's degree in Computer Science, Software Engineering, Information Systems, or related field, or equivalent professional experience.
• Minimum of 8 years of software quality engineering or test automation experience, including at least 2 years testing AI/ML components.
• Experience building automated end-to-end test suites integrated into CI/CD pipelines and defining quality gates.
• Experience leading user-acceptance testing with business or mission users, ideally in federal or regulated environments.
• Strong test automation skills in Python or TypeScript with Playwright, pytest, or Jest, plus API testing tools.
• Knowledge of LLM evaluation techniques, including golden datasets, rubric-based evaluation, and model regression testing.
• Working knowledge of AWS Bedrock and AWS services for testing AI features.
• Familiarity with GitHub, Jira, and CI/CD pipelines; ability to derive tests from use cases.

Responsibilities

• Define the test strategy for AI applications including unit, integration, end-to-end, UAT, security, and AI-behavior testing.
• Design, build, and execute end-to-end and user-acceptance tests across DEV, TST, STG, and PRD.
• Build evaluation harnesses for AI behavior with datasets and rubric-based grading.
• Act as tester in AI development workflow; improve testing agents and skills.
• Automate tests in CI/CD in partnership with platform engineer; own quality gates between environments.
• Validate acceptance criteria with business analyst and real users; capture release evidence.
• Triage, reproduce, and track defects in Jira; drive resolution with engineers.
• Test on AWS-native AI stacks (Bedrock) when required by clients.
• Assess performance, cost, and resilience of AI features under load; report findings.
• Contribute test standards and tooling; coach team on AI evaluation practices.

Skills

Test automation

CI/CD

Python

TypeScript

Playwright

Education

Bachelor's degree in Computer Science or related field

Tools

Playwright

pytest

Jest

GitHub

Jira

Job description

Simple Technology Solutions, LLC is seeking an AI Quality & Test Engineer to own the end-to-end testing of AI applications built on AWS-native services. You will design test strategies, execute UAT and integration tests, and ensure AI outputs meet acceptance criteria in a remote, nationally distributed team.

You will lead automated testing in CI/CD, collaborate with business analysts and users, and help advance QA standards across the STS SDLC, including security and regulatory compliance for