Senior QA and AI Evals Engineer

Ruya AI; California; Híbrido; USD 120.000 - 180.000; Jornada completa

Ventajas ofrecidas por este puesto de trabajo

Relocation assistance

Medical, dental, and vision insurance

Commuter benefits

Paid time off built into the end of年

Leave-of-absence program and flexible/

Descripción de la vacante

Ruya AI is expanding its engineering team and seeks a QA engineer focused on testing AI-enabled platforms. You will design and implement automated test suites across API, UI, and AI evaluation workflows, ensuring reliability for deterministic and non-deterministic outputs.

Collaborating with engineers, you will push for high-signal tests, robust observability, and smooth CI pipelines using Playwright, pytest, and TypeScript in a hybrid U.S. and remote setup.

Formación

• Strong Playwright and API/integration testing experience.
• Experience with pytest and TypeScript test frameworks such as Vitest.
• Understanding of asynchronous and distributed-system failure modes.
• Experience evaluating LLM or other non-deterministic systems.
• Ability to design high-signal test suites rather than maximizing test counts.

Responsabilidades

• Build E2E coverage for investigations, maps, graphs, authentication, and chat.
• Develop API, contract, integration, and component tests across TypeScript and Python services.
• Create Gherkin/BDD scenarios and build Playwright test cases for important UI flows.
• Build AI evaluation datasets, metrics, experiments, and regression gates using Langfuse.
• Test streaming disconnects, replay, cancellation, retries, race conditions, and long-running jobs.
• Validate authorization boundaries and sensitive-data handling.
• Integrate test suites into GitHub Actions and manage flaky-test diagnosis.
• Partner with engineers early to create testable interfaces and useful observability.

Conocimientos

Playwright testing

API testing

Distributed systems understanding

LLM evaluation

Test strategy design

Herramientas

Playwright

pytest

TypeScript

Vitest

Langfuse

Descripción del empleo

Ruya AI is a venture-backed defense-tech company building AI-native intelligent software for sovereign institutions and their affiliated organizations operating in mission-critical, high-stakes environments. Headquartered in Massachusetts (US), with offices across the U.S., Europe, and the Middle East, we are expanding our engineering team with people who demonstrate strong ownership, sound judgment, and the ability to turn complex requirements into reliable products.

The Role

Quant is the team responsible for our multi-modal intelligence agent. Our software is built to surface hidden patterns across disparate data sources, and generate predictive outcomes for the operators who rely on it.

As part of this team, you’re responsible for the quality system behind our platform, covering both conventional software behavior and non-deterministic AI outcomes. Our platform must hold up across deterministic user flows and probabilistic AI outputs alike, staying reliable for the operators who depend on it. You’ll design and build the automated test suites, evaluation pipelines, and release gates behind that reliability.

Core Responsibilities

• Build E2E coverage for investigations, maps, graphs, authentication, and chat.
• Develop API, contract, integration, and component tests across TypeScript and Python services.
• Create Gherkin/BDD scenarios and build Playwright test cases for important UI flows.
• Build AI evaluation datasets, metrics, experiments, and regression gates using Langfuse.
• Test streaming disconnects, replay, cancellation, retries, race conditions, and long-running jobs.
• Validate authorization boundaries and sensitive-data handling.
• Integrate test suites into GitHub Actions and manage flaky-test diagnosis.
• Partner with engineers early to create testable interfaces and useful observability.

Required Qualifications

• Strong Playwright and API/integration testing experience.
• Experience with pytest and TypeScript test frameworks such as Vitest.
• Understanding of asynchronous and distributed-system failure modes.
• Experience evaluating LLM or other non-deterministic systems.
• Ability to design high-signal test suites rather than maximizing test counts.

Preferred Qualifications

Experience with Langfuse, playwright-bdd, fault injection, contract testing, security testing, durable workflows, or model/prompt deployment gates.

Location

In the US, this role is based at our headquarters in Massachusetts, where much of the work takes place. Remote arrangements are also available in California, Massachusetts, Texas, Maryland, Virginia and across the US.

We also hire at our locations in Dubai (UAE), Bosnia, Croatia, and Serbia. Visa sponsorship and relocation assistance are available for our UAE and European locations.

Salary and Benefits

The salary range for this role is highly competitive and benchmarked to market. Total compensation may also include equity through Ruya AI’s employee stock ownership plan (ESOP), a sign-on bonus, and other incentives over time. Final compensation reflects each candidate’s qualifications, experience, skills, certifications, and work location. All offers are contingent on a satisfactory background check and, where applicable, a reference check. Military fellows and part-time employees are not eligible for benefits. Please contact your talent acquisition representative with any questions.

Our benefits are designed to support the health and wellbeing of every Ruyan. We regularly review and refine our offerings based on employee feedback. Below are some of the benefits and perks available to Ruya AI employees:

• Employees (and their eligible dependents) can enroll in medical, dental, and vision insurance
• Commuter benefits
• Relocation assistance
• Paid time off built into the end of each year (subject to team and business needs) and based on location
• A supportive leave-of-absence program, including time off for military service and medical events

Life at Ruya

Growth at Ruya AI isn’t tied to a fixed ladder. We pay attention to what each Ruyan is good at and where they want to go, and we shape roles and opportunities around that. Health and wellbeing benefits are part of this investment, though specifics can vary by region.

We work primarily from our offices because in-person collaboration produces better outcomes, and most teams offer one or two remote days a week. A small number of roles are remote-eligible on an exception basis; employees in those roles must be based in the state where they’re employed, and roles marked onsite require working from an office.

Ruya AI is proud to be an equal opportunity workplace and is an affirmative action employer. We are committed toequal employment opportunity regardless of race, color, ancestry, religion, sex, national origin, sexual orientation, age, marital status, disability, gender identity or Veteran status. If you have a disability or special need that requires accommodation, please let us know.