Senior QA & AI Evaluation Engineer

Ruya AI; California; Hybrid; USD 120,000 - 180,000; Full time

Benefits offered by this job

Relocation assistance

Medical, dental, and vision insurance

Commuter benefits

Paid time off built into the end of年

Leave-of-absence program and flexible/

Job summary

Ruya AI is expanding its engineering team and seeks a QA engineer focused on testing AI-enabled platforms. You will design and implement automated test suites across API, UI, and AI evaluation workflows, ensuring reliability for deterministic and non-deterministic outputs.

Collaborating with engineers, you will push for high-signal tests, robust observability, and smooth CI pipelines using Playwright, pytest, and TypeScript in a hybrid U.S. and remote setup.

Qualifications

• Strong Playwright and API/integration testing experience.
• Experience with pytest and TypeScript test frameworks such as Vitest.
• Understanding of asynchronous and distributed-system failure modes.
• Experience evaluating LLM or other non-deterministic systems.
• Ability to design high-signal test suites rather than maximizing test counts.

Responsibilities

• Build E2E coverage for investigations, maps, graphs, authentication, and chat.
• Develop API, contract, integration, and component tests across TypeScript and Python services.
• Create Gherkin/BDD scenarios and build Playwright test cases for important UI flows.
• Build AI evaluation datasets, metrics, experiments, and regression gates using Langfuse.
• Test streaming disconnects, replay, cancellation, retries, race conditions, and long-running jobs.
• Validate authorization boundaries and sensitive-data handling.
• Integrate test suites into GitHub Actions and manage flaky-test diagnosis.
• Partner with engineers early to create testable interfaces and useful observability.

Skills

Playwright testing

API testing

Distributed systems understanding

LLM evaluation

Test strategy design

Tools

Playwright

pytest

TypeScript

Vitest

Langfuse

Job description

Ruya AI is expanding its engineering team and seeks a QA engineer focused on testing AI-enabled platforms. You will design and implement automated test suites across API, UI, and AI evaluation workflows, ensuring reliability for deterministic and non-deterministic outputs.

Collaborating with engineers, you will push for high-signal tests, robust observability, and smooth CI pipelines using Playwright, pytest, and TypeScript in a hybrid U.S. and remote setup.