Senior Software Development Engineer in Test

Qualia; Oregon; Hybrid; USD 140,000 - 200,000; Full time

Benefits offered by this job

Health plans

401k

Commuter benefits

Parental leave

Flexible time off

Onboarding program

All hands meetings

Job summary

Qualia is hiring a Senior AI/Agentic Quality Infrastructure SDET to build production-grade AI-assisted test infrastructure and agentic validation systems. You’ll design and implement AI-enabled evaluation harnesses, observability platforms, and tooling to validate complex workflows across apps, databases, and CI/CD pipelines.

You will mentor engineers, evolve testing architectures, and collaborate with SRE, Platform, and Engineering teams to deliver scalable, reliable testing solutions in a

Qualifications

• 5+ years as an SDET or software engineer building test infrastructure, developer tooling, or observability systems.
• Strong programming in JavaScript/TypeScript, including backend services and infrastructure.
• Solid fundamentals in system design, distributed systems, APIs, and production ops.
• Experience with CI/CD, release orchestration, test infrastructure, and observability systems.
• Hands-on with LLMs, AI-assisted engineering, or agentic systems including tool calling and multi-step workflows.
• Familiarity with LLM/agent evaluation and probabilistic/non-deterministic testing challenges.
• Understanding of testing pyramid and where validation belongs (unit to agentic layers).
• Mentoring engineers and raising quality bar across teams.

Responsibilities

• Build and maintain test infrastructure, AI/ML evaluation systems, and agentic validation harnesses as production-grade software.
• Design agentic test harnesses enabling AI agents to interact with apps, browsers, APIs, databases, logs, and CI/CD.
• Develop AI/ML evaluation frameworks with deterministic and model-based evaluators and quality signals.
• Incorporate observability/telemetry into automation and agentic infra with logs, metrics, and traces.
• Differentiate product/test/infrastructure/AI failures to speed root-cause analysis.
• Own test automation frameworks and migration paths (e.g., Cypress to Playwright) and scale CI/CD provisioning.
• Apply AI to engineering-quality problems and build guardrails for evidence-backed conclusions.
• Collaborate with software, quality, SRE, and platform teams to build reusable platforms and mentor others.

Skills

SDET experience

JavaScript/TypeScript

System design

CI/CD

LLM/agentic systems

Mentoring

Tools

Playwright

Jest

Postman

Locust

Gatling

JMeter

k6

Jenkins

GitHub Actions

ArgoCD

Spinnaker

Harness

Datadog

OpenTelemetry

MongoDB

Kubernetes

AWS

Google Cloud

Job description

At Qualia, we've built the leading B2B real estate technology that transforms the home buying and selling experience into a simple, secure, and enjoyable process. Our SMB and Enterprise products bring together users from across the real estate ecosystem—homebuyers and sellers, lenders, title and escrow agents, and real estate agents—onto a single shared digital closing platform, providing greater clarity and transparency to real estate transactions. Today, through our business customers across the country, millions of consumers use Qualia to close on homes every year.

WHAT YOU'LL WORK ON

We are hiring pioneering and collaborative engineers to build our next generation of intelligent products. Our quality engineers are conscientious hackers who thrive on collaboration and are adept at driving large-scale projects in the fast-changing domain of artificial intelligence. You will help build the automation, tooling, observability, and quality systems that let us ship complex, AI-driven products with confidence, bringing rigor and trust to a data-rich industry.As a Senior AI/Agentic Quality Infrastructure SDET, you're a software engineer first. You build test infrastructure, agentic validation systems, AI/ML evaluation harnesses, observability platforms, and engineering tooling—not manual test cases or collections of automation scripts. You will help evolve our quality architecture from primarily deterministic automation toward a combination of traditional automation and agentic validation, building systems where AI agents can understand application state, execute workflows, use engineering tools, validate outcomes, investigate failures, and provide evidence for their conclusions.You can take a hard, loosely defined problem and solve it. You're trusted to make technical decisions on frameworks, architecture, evaluation strategy, observability, and validation across every layer of the testing pyramid. You work closely with Engineering and Quality leadership, understand the systems you're testing at a deep technical level, and mentor engineers on building reliable tests, tooling, infrastructure, and AI-assisted validation systems.

RESPONSIBILITIES

• Build and maintain test infrastructure, AI/ML evaluation systems, agentic validation harnesses, release orchestration tooling, observability, and performance testing platforms as production-grade software rather than scripts
• Design agentic test harnesses that let AI agents interact with applications, browsers, APIs, databases, logs, telemetry, and CI/CD systems to execute and validate complex workflows, reasoning about application state rather than relying solely on predefined test scripts
• Build AI/ML evaluation frameworks for validating non-deterministic systems, including deterministic assertions, semantic and model-based evaluation, regression datasets, and human-in-the-loop review, defining measurable quality signals such as correctness, reliability, latency, cost, and hallucination/error rates
• Build observability and telemetry into our automation and agentic infrastructure—structured logs, metrics, distributed tracing, and execution timelines—so engineers can understand why an automated or agentic workflow succeeded or failed
• Develop systems that automatically distinguish product, test, infrastructure, and AI/model reasoning failures, correlating them with logs, traces, deployments, and commits to accelerate root-cause analysis
• Design and build our test automation frameworks, including the migration from Cypress to Playwright, and own core infrastructure such as test environment provisioning, parallelization, and CI/CD integration at scale
• Apply AI to engineering-quality problems including test generation, flaky-test triage, failure clustering, and release-risk analysis, while building guardrails so agent conclusions are backed by observable evidence
• Partner with Software Engineers, Quality Engineers, SRE, and Platform teams to make these capabilities reusable platforms, and mentor engineers on test architecture, observability, and AI-assisted engineering

YOUR BACKGROUND THAT LIKELY MAKES YOU A MATCH

• 5+ years of experience as an SDET or software engineer building test infrastructure, developer tooling, platform engineering, observability, or release engineering systems—not primarily writing test cases or automation scripts
• Strong programming ability in JavaScript/TypeScript, including designing backend services, libraries, frameworks, and production-quality infrastructure from scratch
• Strong software engineering fundamentals including system design, distributed systems, APIs, asynchronous systems, and production operations
• Experience building or scaling CI/CD systems, release orchestration, test infrastructure, or developer platforms, and designing observability/telemetry systems using structured logging, metrics, and distributed tracing
• Hands‑on experience with LLMs, AI‑assisted engineering, or agentic systems beyond simply using an AI coding assistant, including tool/function calling, agent workflows, and multi‑step AI workflows
• Experience or familiarity with LLM and agent evaluation, including evaluation datasets, deterministic and model-based evaluators, and the unique challenges of testing probabilistic/non‑deterministic systems
• Strong understanding of the testing pyramid and experience determining when validation belongs at the unit, integration, API, UI, performance, and agentic layer
• Comfortable mentoring engineers and raising the technical quality bar across teams

THE TECH STACK YOU'LL USE

• JavaScript / Node.js / TypeScript
• Playwright (browser & test automation)
• LLM APIs, tool/function calling, agent frameworks, structured outputs, RAG/context retrieval
• Jest (unit/integration testing)
• Postman / REST Assured (API testing)
• Locust / Gatling / JMeter / k6 (performance testing)
• Jenkins / GitHub Actions / ArgoCD / Spinnaker / Harness (CI/CD & release orchestration)
• Datadog, OpenTelemetry (observability & telemetry)
• MongoDB
• Microservice architecture / Kubernetes
• AWS / Google Cloud

We are a remote‑first development organization that values flexibility in how our teams work. Because of that, while the role is based in Austin, TX, we are open to remote candidates for this position. We have other physical office locations in San Francisco, California, and Concord, New Hampshire.

WHY QUALIA

Qualia is made up of incredibly bright, mission‑driven coworkers who are passionate about using technology to solve real‑world problems—and we're growing quickly. In order to continue building an engaging and dynamic organization, we're committed to giving everyone the support they need to do great work.Our benefits package is designed to allow our team members to be their best selves, both in and out of the workplace. In addition to comprehensive health plans, a 401k program, and commuter benefits, we prioritize family and personal well‑being through professional development, parental leave, and a flexible time off policy. Qualia offers a robust online onboarding program to train new hires, biweekly all hands meetings, and a variety of internal virtual events to keep employees connected.

We believe diverse perspectives and backgrounds are critical to building great technology, and our goal is to cultivate an environment where people feel equally valued and respected.

Qualia is proud to be an equal‑opportunity workplace, and we welcome applicants from all backgrounds regardless of race, color, ancestry, religion, gender identity or expression, sexual orientation, marital status, age, citizenship, socioeconomic status, disability, or veteran status.

By submitting your application, you acknowledge and agree to the collection, processing, and use of your personal information as described in our Employee Data Privacy Notice .#LI-Remote