Senior AI Automation QA Engineer (JS/TS)

Ciklum; United States; Remote; USD 19,000 - 29,000; Full time

Benefits offered by this job

Medical insurance

Certifications

Global projects

Job summary

Ciklum is seeking a Senior Automation QA (JavaScript) Engineer to join our India team on a full-time basis. You will own end-to-end automated testing for frontend and backend components, including long-running workflows, multi-tenant isolation, and non-deterministic AI testing.

You will design and maintain test suites with Playwright and TypeScript, integrate with CI/CD, and mentor mid-level QA engineers. This role demands strong statistical thinking, security-minded testing, and collaboration

Qualifications

• 8+ years of test automation with API testing, and building test frameworks other engineers adopt, not just test cases.
• Hands-on experience testing LLM-powered or other non-deterministic systems statistical assertions, semantic scoring, and managing model variance.
• Experience testing asynchronous, event-driven systems failure injection, idempotency verification, eventual-consistency assertions; Temporal or a similar workflow engine a strong plus.
• Strong CI/CD integration skills (GitHub Actions or equivalent) and the discipline to design release gates.
• Solid SQL and data-validation skills; comfort building synthetic test-data pipelines and measuring dataset quality.
• A security-testing and adversarial mindset; familiarity with prompt-injection techniques or a strong AppSec testing background with willingness to specialize fast.
• Demonstrated judgment on what "tested enough" means under milestone pressure, and the ability to defend quality evidence to non-engineering stakeholders.
• Strong proficiency in Playwright and TypeScript is mandatory for building automated test suites.
• QA Automation Core: Extensive experience in building automated test frameworks (Playwright/Typescript-based) and integrating them into high-security CI/CD pipelines.
• AI/LLM Literacy: Deep understanding of Large Language Model behaviors, prompt sensitivity, and the common failure modes of RAG systems.
• Analytical Mindset: Strong capability in statistical analysis to interpret evaluation scores and provide actionable feedback to AI Engineers for prompt tuning

Responsibilities

• Design, build, and maintain automated test suites for the platform, Frontend and Backend, including contract tests that protect the agent-developer experience as the SDK evolves
• Build automated tests for long-running Temporal workflows including failure injection (worker kill, provider outage, retry storms) to verify durable-execution guarantees such as zero lost runs and correct compensation/saga behavior
• Automate multi-tenant isolation verification cross-tenant data access, configuration bleed, and cost-attribution correctness executed per release as a platform acceptance criterion
• Verify agent execution sandboxing and egress controls through negative testing of disallowed tool calls and network destinations
• AI-Specific & Non-deterministic Testing
• Design test strategies for LLM-driven behavior: statistical assertions over repeated runs, semantic-similarity scoring, confidence-threshold validation, and flakiness quarantine
• Build and continuously extend adversarial test corpora document-borne prompt injection in lease PDFs, tool-call hijack, and data-exfiltration probes aligned to OWASP LLM Top 10 and recognized red-team taxonomies
• Verify Human-in-the-Loop escalation behavior confidence thresholds fire correctly and no gated/irreversible action completes without authorization
• Partner with the evaluation function on golden-dataset-driven checks, contributing test infrastructure, synthetic-data pipelines, and CI integration that make agent quality measurable and repeatable
• Performance, Reliability & NFR Verification
• Automate verification of platform NFRs invocation overhead (p95), concurrency targets, queue backpressure behavior, and LLM provider failover time and surface trends per release
• Use tracing data (Langfuse / OpenTelemetry spans) as a first-class test asset: assert on trace completeness, token/cost-accounting accuracy, and alerting behavior for anomalous runs
• Process, Evidence & Collaboration
• Wire test suites into CI/CD as hard release gates with automated, per-metric regression flagging on pull requests
• Produce auditable, reproducible test-evidence packs that feed directly into client milestone sign-off
• Embed with AI and platform engineers from the design stage making acceptance criteria, testability, and observability inputs to architecture, not afterthoughts
• Maintain test environments, mocked LLM/provider layers, and synthetic data generators that keep test runs fast, deterministic where possible, and affordable
• Mentor mid-level QA engineers, codify AI testing patterns into reusable internal frameworks, and contribute to Ciklum's quality engineering practice and AI Academy

Skills

Playwright

TypeScript

CI/CD

API testing

Test automation

Tools

SQL

Job description

Ciklum is seeking a Senior Automation QA (JavaScript) Engineer to join our India team on a full-time basis. You will own end-to-end automated testing for frontend and backend components, including long-running workflows, multi-tenant isolation, and non-deterministic AI testing.

You will design and maintain test suites with Playwright and TypeScript, integrate with CI/CD, and mentor mid-level QA engineers. This role demands strong statistical thinking, security-minded testing, and collaboration