Staff Quality Engineer: AI & Agentic Systems Lead

United States Digital Space LLC; United States; Remote; USD 136,000 - 187,000; Full time

Benefits offered by this job

Competitive salary

401k/retirement savings

Paid time off

Health benefits from day one

Remote work support

Job summary

United States Digital Space LLC is seeking a Staff Quality Engineer to lead quality strategy for AI features. You will design evaluation frameworks, build scalable automated tests, and set quality metrics across non-deterministic AI systems.

As the senior QE, you will mentor the QE org, influence cross-functional teams, and ensure testability from design through release in a remote-friendly environment across North America.

Qualifications

• 7+ years of professional software quality and test automation experience.
• Experience with API & UI test automation frameworks such as Playwright, Cypress, Jest, or JUnit.
• Fluency in at least one of TypeScript, JavaScript, Python, or Java with preference for TypeScript.
• Experience testing AI/agentic systems (evals, prompt/regression testing, output validation) and using AI to scale automation.
• Experience with CI/CD tools like Jenkins, Github Actions, or CircleCI.
• Strong understanding of scalable system design for web-based APIs and backend processing systems.
• Proven ability to define test case design standards and coverage strategy and own test case management (e.g. TestRail).
• Knowledge of React, including adding data-testids.

Responsibilities

• Own and drive the end-to-end quality strategy for AI features from test architecture through release readiness.
• Design evaluation frameworks for non-deterministic systems and guardrail testing.
• Develop scalable automated test suites addressing functional, API, UI, and regression testing in CI/CD.
• Establish quality metrics (accuracy, latency, cost, hallucination rates).
• Collaborate with Engineering, Product, ML/Data Science to align requirements and definitions of done.
• Mentor QE team, promote best practices, and raise testing quality for AI systems company-wide.
• Engage in technical planning, code reviews, and architectural decisions emphasizing testability.
• Provide timely feedback on test results to stakeholders.

Skills

Test automation

AI testing

Technical leadership

CI/CD practices

Quality strategy

Tools

Playwright

Cypress

Jest

JUnit

TypeScript

JavaScript

Python

Java

Jenkins

GitHub Actions

CircleCI

Job description

United States Digital Space LLC is seeking a Staff Quality Engineer to lead quality strategy for AI features. You will design evaluation frameworks, build scalable automated tests, and set quality metrics across non-deterministic AI systems.

As the senior QE, you will mentor the QE org, influence cross-functional teams, and ensure testability from design through release in a remote-friendly environment across North America.