Senior AI QA Automation Engineer

Jobgether; United States; Remote; USD 120,000 - 180,000; Full time

Benefits offered by this job

Remote work flexibility

Global collaboration

Professional development

Job summary

Jobgether is seeking a senior QA engineer focused on automated testing for AI-driven platforms. The role emphasizes scalable test frameworks across frontend, backend, APIs, and contracts, with strong emphasis on LLM testing, security, and reliability in multi-tenant SaaS environments.

You will implement testing strategies for nondeterministic AI systems and champion observability practices. The position requires 8+ years of experience, proficiency in Playwright and TypeScript, and hands-on work

Qualifications

• 8+ years of experience in test automation, APIs, and reusable frameworks.
• Strong proficiency in Playwright and TypeScript; CI/CD integration.
• Experience testing LLM-powered or nondeterministic systems with statistical/assertion methods.
• Knowledge of asynchronous/event-driven architectures and failure-injection testing.
• SQL skills and data validation for synthetic data pipelines.
• Security-focused testing incl. adversarial testing and prompt-injection scenarios.

Responsibilities

• Design, build, and maintain scalable automated test frameworks for frontend, backend, APIs, and contracts.
• Create testing strategies for AI/LLM systems including semantic scoring and confidence thresholds.
• Test long-running, asynchronous workflows with failure injection, retries, and idempotency checks.
• Ensure cross-tenant security, sandboxing, access controls, and cost attribution validation.
• Develop adversarial test datasets to identify prompt injection and data exfiltration risks.
• Validate human-in-the-loop escalation and ensure irreversible actions require authorization.
• Collaborate on golden datasets, synthetic data pipelines, and repeatable eval infra.
• Automate latency, concurrency, and provider failover testing; monitor performance trends.

Skills

Test automation

Playwright

TypeScript

AI/LLM testing

Asynchronous systems

CI/CD

SQL/data validation

Security testing

Observability

Multi-tenant SaaS

Evaluation & analytics

Communication & reporting

Tools

GitHub Actions

OpenTelemetry

Langfuse

LangSmith

Promptfoo

DeepEval

Arize Phoenix

Job description

Jobgether is seeking a senior QA engineer focused on automated testing for AI-driven platforms. The role emphasizes scalable test frameworks across frontend, backend, APIs, and contracts, with strong emphasis on LLM testing, security, and reliability in multi-tenant SaaS environments.

You will implement testing strategies for nondeterministic AI systems and champion observability practices. The position requires 8+ years of experience, proficiency in Playwright and TypeScript, and hands-on work