Senior Automation QA (JS) Engineer

Jobgether; United States; Remote; USD 120,000 - 180,000; Full time

Benefits offered by this job

Remote work flexibility

Global collaboration

Professional development

Job summary

Jobgether is seeking a senior QA engineer focused on automated testing for AI-driven platforms. The role emphasizes scalable test frameworks across frontend, backend, APIs, and contracts, with strong emphasis on LLM testing, security, and reliability in multi-tenant SaaS environments.

You will implement testing strategies for nondeterministic AI systems and champion observability practices. The position requires 8+ years of experience, proficiency in Playwright and TypeScript, and hands-on work

Qualifications

• 8+ years of experience in test automation, APIs, and reusable frameworks.
• Strong proficiency in Playwright and TypeScript; CI/CD integration.
• Experience testing LLM-powered or nondeterministic systems with statistical/assertion methods.
• Knowledge of asynchronous/event-driven architectures and failure-injection testing.
• SQL skills and data validation for synthetic data pipelines.
• Security-focused testing incl. adversarial testing and prompt-injection scenarios.

Responsibilities

• Design, build, and maintain scalable automated test frameworks for frontend, backend, APIs, and contracts.
• Create testing strategies for AI/LLM systems including semantic scoring and confidence thresholds.
• Test long-running, asynchronous workflows with failure injection, retries, and idempotency checks.
• Ensure cross-tenant security, sandboxing, access controls, and cost attribution validation.
• Develop adversarial test datasets to identify prompt injection and data exfiltration risks.
• Validate human-in-the-loop escalation and ensure irreversible actions require authorization.
• Collaborate on golden datasets, synthetic data pipelines, and repeatable eval infra.
• Automate latency, concurrency, and provider failover testing; monitor performance trends.

Skills

Test automation

Playwright

TypeScript

AI/LLM testing

Asynchronous systems

CI/CD

SQL/data validation

Security testing

Observability

Multi-tenant SaaS

Evaluation & analytics

Communication & reporting

Tools

GitHub Actions

OpenTelemetry

Langfuse

LangSmith

Promptfoo

DeepEval

Arize Phoenix

Job description

Accountabilities:

• Test Automation Framework Development: Design, build, and maintain scalable automated test frameworks using Playwright and TypeScript. Develop comprehensive test suites covering frontend applications, backend services, APIs, and contract compatibility to ensure reliable platform behavior as products evolve.
• AI and LLM Quality Assurance: Create specialized testing strategies for nondeterministic AI systems, including statistical assertions across repeated runs, semantic similarity evaluation, confidence-threshold validation, and techniques for distinguishing model variability from genuine software regressions.
• Workflow Reliability and Resilience Testing: Validate long-running, asynchronous, and event-driven workflows through failure injection, retry scenarios, worker termination, provider outages, idempotency checks, and eventual-consistency assertions. Verify durable execution and appropriate compensation mechanisms during failures.
• Multi-Tenant Security and Isolation: Develop automated tests to identify cross-tenant data exposure, configuration leakage, and inaccurate cost attribution. Validate execution sandboxing, network restrictions, and access controls to ensure platform security and tenant isolation.
• Adversarial and AI Security Testing: Build and maintain adversarial test datasets to identify prompt injection, tool-call manipulation, unauthorized actions, and potential data exfiltration. Align testing practices with recognized AI security frameworks, including the OWASP Top 10 for LLM Applications.
• Human-in-the-Loop Validation: Verify that confidence thresholds trigger the appropriate escalation procedures and that restricted or irreversible actions cannot proceed without the required human authorization.
• Evaluation Infrastructure and Test Data: Collaborate with AI evaluation teams to build golden-dataset-driven validation processes, synthetic-data generation pipelines, and repeatable evaluation infrastructure. Ensure datasets are reliable, measurable, and suitable for ongoing regression testing.
• Performance and Non-Functional Testing: Automate verification of latency, invocation overhead, concurrency, queue backpressure, and AI provider failover times. Analyze performance trends across releases and identify opportunities to improve system efficiency and resilience.
• Observability and Trace Validation: Use tools such as Langfuse and OpenTelemetry to validate trace completeness, token usage, cost accounting, and anomaly detection. Incorporate observability data into automated testing and troubleshooting workflows.
• CI/CD Integration and Release Quality Gates: Integrate automated test suites into CI/CD pipelines using GitHub Actions or equivalent tools. Establish mandatory release gates, automate regression detection, and flag metric-level quality issues during pull requests and deployments.
• Quality Evidence and Stakeholder Collaboration: Produce reproducible, auditable test reports and evidence packages to support client milestone approvals. Communicate testing outcomes, risks, and quality assessments clearly to technical and non-technical stakeholders.
• Architecture and Testability: Partner with AI and platform engineers from the design stage to define acceptance criteria, improve testability, and incorporate observability and quality considerations into system architecture.
• Environment Management and Mentorship: Maintain test environments, mocked AI providers, and synthetic data generators to enable efficient and cost-effective testing. Mentor mid-level QA engineers, develop reusable testing frameworks, and contribute to shared quality engineering practices.

Requirements:

• Professional Experience: At least 8 years of experience in test automation, API testing, and developing reusable test frameworks adopted by engineering teams.
• Automation Expertise: Strong proficiency in Playwright and TypeScript is mandatory, with demonstrated experience designing comprehensive automated test suites and integrating them into secure, reliable CI/CD pipelines.
• AI and LLM Testing: Hands-on experience testing LLM-powered or other nondeterministic systems, including statistical assertions, semantic scoring, prompt sensitivity, model variability, and evaluation of common Retrieval-Augmented Generation (RAG) failure modes. Candidates should be able to demonstrate practical examples of these testing approaches.
• Asynchronous Systems: Experience testing event-driven architectures, distributed workflows, and background processing systems. Strong knowledge of failure injection, idempotency, retries, and eventual consistency is required; experience with Temporal or a comparable workflow engine is a significant advantage.
• CI/CD and Release Management: Proven ability to integrate automated testing into continuous integration and delivery pipelines, preferably using GitHub Actions or equivalent tools. Experience defining release gates and making informed quality decisions under tight deadlines is essential.
• Database and Data Validation: Solid SQL skills, experience validating data integrity, and the ability to develop synthetic test-data pipelines and assess dataset quality.
• Security Testing: A security-focused mindset with experience in adversarial testing, application security, or prompt-injection testing. Candidates should be comfortable identifying vulnerabilities and developing effective negative test scenarios.
• Analytical and Evaluation Skills: Strong statistical and analytical capabilities, including interpreting evaluation scores, identifying behavioral patterns, and providing actionable feedback to AI engineers for model and prompt improvements.
• Quality Judgment and Communication: Demonstrated ability to determine acceptable quality thresholds under delivery pressure, assess technical risks, and defend testing evidence and release recommendations to non-engineering stakeholders.
• Performance Testing: Familiarity with performance and load-testing tools such as k6 or Locust is an advantage, particularly for validating concurrency, latency, and high-volume background processing.
• AI Observability and Evaluation Tools: Exposure to tools such as Langfuse, LangSmith, Promptfoo, DeepEval, or Arize Phoenix is desirable.
• Multi-Tenant SaaS Experience: Previous experience testing tenant isolation, access controls, and data separation in multi-tenant SaaS platforms is beneficial.
• Compliance and Accessibility: Experience producing compliance evidence for frameworks or regulations such as SOC 2 and GDPR, or performing accessibility testing against Section 508 or WCAG standards, is an advantage.
• Domain Knowledge: Familiarity with commercial real estate workflows, lease accounting, common area maintenance (CAM) reconciliation, document extraction, OCR, or document layout parsing is beneficial.
• Advanced Testing Practices: Experience using LLMs to generate test cases, adversarial payloads, or synthetic documents is an advantage. Familiarity with qTest for test management is also desirable.
• Collaboration and Mentorship: Strong communication, problem-solving, and teamwork skills, with the ability to collaborate across engineering disciplines, share expertise, and promote consistent quality practices.
• Work Arrangement: Willingness to work remotely from India and collaborate effectively with distributed international teams.

Benefits:

• Remote Work Flexibility: Work remotely from India as part of a globally distributed professional community.
• Collaborative Environment: Join a supportive, open-door culture that encourages knowledge sharing, teamwork, and continuous improvement.
• Professional Development: Access internal meetups, conferences, workshops, language courses, and Udemy learning resources.
• Company-Funded Certifications: Opportunities to strengthen your technical expertise through certifications sponsored by the employer.
• Career Growth: Work on large-scale, high-impact projects and explore internal mobility opportunities across different industries and technology domains.
• Cutting-Edge Technology: Gain hands-on experience with AI-powered applications, advanced automation frameworks, and modern quality engineering practices.
• Healthcare Coverage: Company-paid medical insurance.
• Mental Health Support: Access to resources designed to support employee well-being.
• Financial and Legal Assistance: Access to financial and legal consultation services.
• Global Collaboration: Work alongside experienced engineers and technology professionals on innovative solutions with international reach.