Senior QA Engineer

XY.ai Inc.; United States; Hybrid; USD 120,000 - 180,000; Full time

Benefits offered by this job

Equity

Job summary

XY.AI Labs is building an AI-powered, fully-automated quality and release platform in a health-tech startup. The role emphasizes hands-on engineering across Python backend and Node/Next.js frontend, with GitHub Actions pipelines and containerized environments.

You will design prompt-stress suites, living dashboards, and governance for progressive releases while mentoring teammates and shaping our QA strategy as we scale from day 0 to unicorn.

Qualifications

• Proven design of end-to-end automation frameworks and zero-downtime releases.
• Experience mentoring dev teams in quality practices.
• Hands-on testing of ML/LLM features, prompt engineering, and bias checks.

Responsibilities

• Build and maintain test infrastructure in Python (backend) and Node/Next.js (frontend).
• Architect GitHub Actions pipelines for unit, API, UI, and performance tests.
• Implement container-first test environments (Docker/Kubernetes) and IaC for staging and releases.
• Create dashboards showing coverage, MTTR, and AI performance drift for leadership.

Skills

QA automation

Full-stack engineering

LLM evaluation

Prompt engineering

Mentoring

Tools

Python

Node.js

Next.js

Pydantic

GitHub Actions

Selenium

Playwright

Docker

Kubernetes

Job description

XY.AI Labs is a venture-backed startup on a mission to remove friction from healthcare. Our agentic AI platform combines purpose-built agents and intelligence layers to streamline complex operational workflows—giving healthcare providers back what matters most: time for care.

About the Role

Build an AI-powered, fully-automated quality, evaluation, and release platform at a rocket-ship health-tech startup to support our rapidly evolving AI Agent orchestration platform.

You'll contribute to shaping our QA strategy while driving hands-on execution—designing self-healing test harnesses, spinning up LLM-driven evaluation pipelines, and weaving everything into a one-click GitHub Actions release flow.

We're hunting for a 10x, QA-obsessed full-stack engineer who already wields AI tools (Copilot, auto-test agents, code-gen) to amplify output and who thrives on turning blank slates into world-class systems. This is your canvas to experiment, take smart risks, and help scale quality from day 0 to Unicorn.

Key Responsibilities

AI-Enhanced Full-Stack Test Automation

• Build and maintain test infrastructure in Python (back-end) and Node/Next.js (front-end), instrumented with Pydantic models for type-safe data flows.
• Architect GitHub Actions pipelines that run unit, API, UI (Selenium/Playwright) and performance suites on every PR, auto-deploying to GCP or Azure test environments.
• Implement container-first test environments (Docker/K8s) and infrastructure-as-code for repeatable staging and blue/green releases.
• Craft living dashboards that surface coverage, MTTR, and AI performance drift metrics to Engineering & Product leadership.

LLM Evaluation & AI Quality

• Design prompt-stress suites, reference-answer graders, and automated hallucination/bias checks to guarantee safe, deterministic agent behavior.
• Stay tool-agnostic so we can adopt or sunset frameworks quickly as the LLM landscape evolves.

Autonomous Release Governance

• Act as final quality gatekeeper—own the go/no-go call, publish release notes, and champion progressive rollout strategies (canary, feature flags).

Growing the QA Practice

• Contribute to shaping our QA strategy and help establish peer code-review standards that propagate a culture of testability across all services.
• As the team grows, help onboard and mentor new QA team members.

What We're Looking For

1

Full-Stack Automation Mastery

2

Proven design of end-to-end automation frameworks, shift-left testing, and zero-downtime releases. Experience mentoring dev teams in quality practices.

3

AI / LLM Evaluation

Hands-on testing of ML or LLM features—prompt engineering, response grading, drift detection. Understanding of AI safety, bias, and data privacy.

4

Comfortable owning a charter, setting priorities with VP Eng/Product, and communicating trade-offs in plain English. Startup or 0→1 experience is strongly preferred.

You Might Be a Great Fit If...

You Have:

• Builder-Owner Mentality — you see constraints as creative fuel and bias toward shipping.
• Data-Driven Experimenter — you design hypotheses, measure, iterate.
• Mentor at Heart — eager to uplevel peers and cultivate a blameless, high-trust culture.

The Growth Path

Join as a key early member of the QA function, with opportunities to grow into senior technical or leadership roles as the team scales. You'll have the chance to explore emerging LLM-Ops tooling and help position XY as an industry pioneer in AI quality engineering.

Why Join Us

Build quality infrastructure from scratch at a production-grade AI agent system

Build in a space with real-world urgency — billing errors = lost care

Pioneer LLM evaluation techniques and define industry benchmarks

Collaborate with a world-class team — we ship fast

Your quality gates will protect healthcare operations used every single day

Compensation

High-growth opportunity with equity in an early-stage startup.

XY.AI Labs is an equal opportunity employer. We celebrate diversity and are committed to creating an inclusive environment for all employees.