Senior QA Automation Engineer, AI Agentic Systems (Remote)

GCS Recruitment; United States; Remote; USD 120,000 - 170,000; Full time

Job summary

GCS Recruitment seeks a Senior QA Automation Engineer focused on AI and agentic systems. You will own testing frameworks for non-deterministic AI, tool selection strategies, and grounding verification, while also validating API, performance, and data integrity across live production.

You will collaborate with engineering to design regression suites, implement automated evaluation using Lua/LangSmith-like tooling, and ensure quality gates before releases. Remote role with senior ownership.

Qualifications

• 5+ years in QA automation or quality engineering, with at least 2 years focused on ML models, LLM apps, or AI agents.
• Ability to define pass/fail criteria for non-deterministic outputs and communicate confidence levels to leadership.
• A senior individual contributor who scopes and builds testing frameworks independently.
• Strong collaboration with engineering and product teams and quick system onboarding.

Responsibilities

• Lead Agentic System Testing focusing on tool-use trajectories, grounding, and honesty in outputs.
• Develop automated frameworks for AI evaluation and prompt/model regression suites.
• Design and run performance and API tests, ensuring secure gateway validation and RBAC compliance.
• Implement continuous evaluation in production with monitoring, alerts, and incident response.
• Maintain test reports and artifacts and link results to requirements IDs.

Skills

QA automation

Independent operator

Collaboration

Test design

Python

API testing

Performance testing

CI/CD testing

Tools

Git/GitLab

Selenium/Playwright

TestRail

Allure

LangSmith/Langfuse

OpenTelemetry

Job description

GCS Recruitment seeks a Senior QA Automation Engineer focused on AI and agentic systems. You will own testing frameworks for non-deterministic AI, tool selection strategies, and grounding verification, while also validating API, performance, and data integrity across live production.

You will collaborate with engineering to design regression suites, implement automated evaluation using Lua/LangSmith-like tooling, and ensure quality gates before releases. Remote role with senior ownership.