Remote AI QA & Test Engineer for LLMs

simpletechnologysolutions; Washington (District of Columbia); Remote; USD 140,000 - 180,000; Full time

Benefits offered by this job

HUBZone incentives

Job summary

Simple Technology Solutions is seeking a Senior QA/Test Automation Engineer to lead testing for AI-enabled government solutions. The role emphasizes building automated end-to-end tests, AI evaluation harnesses, and integrating tests into CI/CD pipelines.

Remote work with occasional travel, preference for DC Metro region and HUBZone incentives. The ideal candidate will have 8+ years in software quality engineering, strong Python/TypeScript automation skills, and familiarity with LLM evaluation

Qualifications

• Bachelor’s degree or equivalent professional experience.
• 8+ years of software quality engineering or test automation
• Experience testing applications built on large language models or other AI/ML components
• Experience leading user-acceptance testing with business or mission users, ideally in federal or regulated environment
• Strong test automation skills in Python or TypeScript with Playwright, pytest, or Jest
• Practical knowledge of LLM evaluation techniques, including golden datasets and rubric-based evaluation

Responsibilities

• Define the test strategy for AI applications across DEV, TST, STG, and PRD.
• Design, build, and execute end-to-end and user-acceptance tests.
• Build evaluation harnesses for AI behavior and safety checks.
• Automate tests in the CI/CD pipeline and maintain quality gates.
• Triage, reproduce, and track defects in Jira and coordinate with engineers.

Skills

Python

Test automation

Playwright

CI/CD

LLM testing

Education

Bachelor's degree in CS or related field

Equivalent professional experience

Tools

Jira

Playwright

pytest

Jest

Bedrock

Job description

Simple Technology Solutions is seeking a Senior QA/Test Automation Engineer to lead testing for AI-enabled government solutions. The role emphasizes building automated end-to-end tests, AI evaluation harnesses, and integrating tests into CI/CD pipelines.

Remote work with occasional travel, preference for DC Metro region and HUBZone incentives. The ideal candidate will have 8+ years in software quality engineering, strong Python/TypeScript automation skills, and familiarity with LLM evaluation