Senior AI Test Engineer – Eval & Observability

Vibehackers; Austin (TX); On-site; USD 120,000 - 180,000; Full time

Benefits offered by this job

Curative Health Plan (100% employer‑en

$0 copays and $0 deductibles

Preventive and primary care

Chronic condition programs

Maternity and family planning support

Life and disability coverage

Flexible spending accounts

11 paid company holidays

401(k) for full-time employees

Paid parental leave

Job summary

Curative is seeking a Senior Software Test Engineer to build eval infrastructure and perform hands-on exploratory testing for AI agents, enabling rapid, high-quality delivery. This role blends automation, production forensics, and collaboration with business stakeholders to measure agent behavior, latency, and cost.

You will work with Python and TypeScript, developing dashboards and observability tooling to support reliable agent performance and governance.

Qualifications

• 5+ years in software quality, testing, or engineering with automation and exploratory testing.
• Strong fluency with LLM evaluation: golden datasets, programmatic graders, LLM-as-judge approaches.
• Hands-on experience with AI agents and failure modes (silent drift, tool misuse).
• Deep debugging skills: logs, traces, production forensics; Python or TypeScript debugging.
• Strong stakeholder communication and ability to extract requirements from non-technical partners.
• Comfort operating without mature processes and reviewing AI-generated code diffs.
• Excellent written communication and pragmatic quality judgment.

Responsibilities

• Design, build, and maintain the eval platform with datasets, graders, and CI integration.
• Perform hands-on exploratory testing across member, operations, and provider workflows.
• Validate delivered product matches stakeholder needs by direct collaboration with business teams.
• Develop production signals, dashboards, and failure taxonomies; reproduce incidents and diagnose root causes.
• Make pragmatic product and prioritization decisions under ambiguity; act as de facto PQO when needed.

Skills

Test Automation

Exploratory Testing

Debugging

Production Forensics

Observability

CI/CD Integration

Regression Testing

Stakeholder Communication

Prioritization

Product Judgment

Written Communication

Domain Expertise

Tools

Python

TypeScript

Claude Code

Cursor

LangSmith

Braintrust

Arize

Job description

Curative is seeking a Senior Software Test Engineer to build eval infrastructure and perform hands-on exploratory testing for AI agents, enabling rapid, high-quality delivery. This role blends automation, production forensics, and collaboration with business stakeholders to measure agent behavior, latency, and cost.

You will work with Python and TypeScript, developing dashboards and observability tooling to support reliable agent performance and governance.