Senior AI QA & Evaluation Engineer

Ruya; Maryland; Hybrid; USD 120,000 - 180,000; Full time

Benefits offered by this job

Health insurance

Dental coverage

Vision coverage

Commuter benefits

Relocation assistance

Paid time off

Job summary

Ruya AI is building AI-native software for sovereign institutions with a focus on reliability across deterministic and probabilistic AI outputs. The Quant team develops the quality system, automated test suites, evaluation pipelines, and release gates for a robust platform.

The role emphasizes end-to-end testing across multiple services, API and UI tests, and preparing regression gates. Remote-ready with hybrid options in several US states, including MD, CA, MA, TX, and VA.

Qualifications

• Strong Playwright and API/integration testing experience.
• Experience with pytest and TypeScript test frameworks such as Vitest.
• Understanding of asynchronous and distributed-system failure modes.
• Experience evaluating LLM or other non-deterministic systems.
• Ability to design high-signal test suites rather than maximizing test counts.

Responsibilities

• Build E2E coverage for investigations, maps, graphs, authentication, and chat.
• Develop API, contract, integration, and component tests across TypeScript and Python services.
• Create Gherkin/BDD scenarios and build Playwright test cases for important UI flows.
• Build AI evaluation datasets, metrics, experiments, and regression gates using Langfuse.
• Test streaming disconnects, replay, cancellation, retries, race conditions, and long-running jobs.
• Validate authorization boundaries and sensitive-data handling.
• Integrate test suites into GitHub Actions and manage flaky-test diagnosis.
• Partner with engineers early to create testable interfaces and useful observability.

Skills

Playwright testing

pytest

Vitest

distributed systems

LLM eval

Tools

Langfuse

Playwright

GitHub Actions

Job description

Ruya AI is building AI-native software for sovereign institutions with a focus on reliability across deterministic and probabilistic AI outputs. The Quant team develops the quality system, automated test suites, evaluation pipelines, and release gates for a robust platform.

The role emphasizes end-to-end testing across multiple services, API and UI tests, and preparing regression gates. Remote-ready with hybrid options in several US states, including MD, CA, MA, TX, and VA.