Senior Software Test Engineer

Vibehackers; Austin (TX); On-site; USD 120,000 - 180,000; Full time

Benefits offered by this job

Curative Health Plan (100% employer‑en

$0 copays and $0 deductibles

Preventive and primary care

Chronic condition programs

Maternity and family planning support

Life and disability coverage

Flexible spending accounts

11 paid company holidays

401(k) for full-time employees

Paid parental leave

Job summary

Curative is seeking a Senior Software Test Engineer to build eval infrastructure and perform hands-on exploratory testing for AI agents, enabling rapid, high-quality delivery. This role blends automation, production forensics, and collaboration with business stakeholders to measure agent behavior, latency, and cost.

You will work with Python and TypeScript, developing dashboards and observability tooling to support reliable agent performance and governance.

Qualifications

• 5+ years in software quality, testing, or engineering with automation and exploratory testing.
• Strong fluency with LLM evaluation: golden datasets, programmatic graders, LLM-as-judge approaches.
• Hands-on experience with AI agents and failure modes (silent drift, tool misuse).
• Deep debugging skills: logs, traces, production forensics; Python or TypeScript debugging.
• Strong stakeholder communication and ability to extract requirements from non-technical partners.
• Comfort operating without mature processes and reviewing AI-generated code diffs.
• Excellent written communication and pragmatic quality judgment.

Responsibilities

• Design, build, and maintain the eval platform with datasets, graders, and CI integration.
• Perform hands-on exploratory testing across member, operations, and provider workflows.
• Validate delivered product matches stakeholder needs by direct collaboration with business teams.
• Develop production signals, dashboards, and failure taxonomies; reproduce incidents and diagnose root causes.
• Make pragmatic product and prioritization decisions under ambiguity; act as de facto PQO when needed.

Skills

Test Automation

Exploratory Testing

Debugging

Production Forensics

Observability

CI/CD Integration

Regression Testing

Stakeholder Communication

Prioritization

Product Judgment

Written Communication

Domain Expertise

Tools

Python

TypeScript

Claude Code

Cursor

LangSmith

Braintrust

Arize

Job description

Explicitly requires vibe coding skills — the team uses AI coding tools (Claude Code, Cursor) as the primary development surface and relies on AI for evals and graders.

About the Role

Senior Software Test Engineer to build eval infrastructure and do hands-on exploratory testing for AI agents, enabling rapid, high-quality delivery by combining automation, production forensics, and close partnership with business stakeholders.

Job Description

Role

Curative is hiring a Senior Software Test Engineer to drive quality and measurement for AI agents and production services. This role combines building eval and observability infrastructure with hands-on exploratory testing, production forensics, and close collaboration with business teams.

Key Responsibilities

• Design, build, and maintain the eval platform: harnesses, datasets, graders, and CI integration to measure agent behavior (task success, tool-use correctness, drift, latency, cost).
• Perform hands-on exploratory testing across member, operations, and provider workflows, including unusual and edge paths.
• Validate that the delivered product matches stakeholder needs by working directly with business teams and extracting real workflows.
• Develop production signal, dashboards, and failure taxonomies; reproduce incidents, diagnose root causes, and fix or hand off precise remediation steps.
• Make pragmatic product and prioritization decisions under ambiguity; act as a de facto product/quality owner when needed.

Requirements

• 5+ years in software quality, testing, or engineering with experience in both automation and exploratory testing.
• Strong fluency with LLM evaluation: golden datasets, programmatic graders, LLM-as-judge approaches, and regression suites for prompts and agent loops.
• Hands-on experience with AI agents and common failure modes (silent drift, tool misuse, compounding errors).
• Deep debugging skills: logs, traces, queries, and production forensics, with ability to fix issues in Python or TypeScript.
• Strong stakeholder communication and the ability to extract requirements from non-technical partners.
• Comfort working without mature processes, making fast trade-offs, and reviewing AI-generated code diffs.
• Excellent written communication and pragmatic quality judgment.

Preferred

• Experience building eval or observability infrastructure from scratch.
• Familiarity with LLM observability tooling (LangSmith, Braintrust, Arize) or equivalent homegrown solutions.
• Experience defining agent guardrails and permissible unsupervised behaviors.

Why this role

You will be close to the business and own both the testing infrastructure and the product outcomes. The role offers wide responsibility across eval infrastructure, hands-on testing, production stakes, and direct collaboration with product and operations.

• Curative Health Plan (100% employer-covered medical premiums for you and 50% coverage for dependents on the base plan)
• $0 copays and $0 deductibles (with completion of Baseline Visit)
• Preventive and primary care, mental health support, one-on-one care navigation, chronic condition programs
• Maternity and family planning support
• Employer-provided life and disability coverage with supplemental options
• Flexible spending accounts
• Generous PTO policy and 11 paid company holidays
• 401(k) for full-time employees
• Up to 8–12 weeks paid parental leave (role eligibility-dependent)

Python TypeScript Claude Code Cursor LangSmith Braintrust Arize

Skills

Test Automation Exploratory Testing Debugging Production Forensics Observability CI/CD Integration Regression Testing Stakeholder Communication Prioritization Product Judgment Written Communication Domain Expertise

Experience Level

Senior

Employment Type

Full-time

• 100% employer-covered medical premiums for employee (50% for dependents on base plan)
• $0 copays and $0 deductibles (with completion of Baseline Visit)
• Preventive and primary care
• Chronic condition programs (diabetes, weight, hypertension)
• Maternity and family planning support
• Employer-provided life and disability coverage (with supplemental options)
• Flexible spending accounts
• 11 paid annual company holidays
• 401K for full-time employees
• Up to 8–12 weeks paid parental leave (based on role eligibility)