Senior AI Systems Quality Engineer (Platform & Guardrails)

The Blue Venture Fund; United States; Remote; USD 150,000 - 230,000; Full time

Benefits offered by this job

Unlimited paid time off

Work from anywhere

Health coverage

Equity for every employee

Growth-focused environment

Home office setup allowance

Monthly cell phone allowance

Job summary

Abacus Insights is building AI-powered platforms that transform critical healthcare decisions. The Senior AI Systems Quality Engineer ensures production-ready, trustworthy AI systems by embedding guardrails, evaluation signals, and safe-failure behavior from design to deployment.

You will lead the development of a Databricks‑native AI testing platform and collaborate with AI engineers and delivery teams to define robust evaluation pipelines and quality gates.

Qualifications

• 7+ years in software engineering, backend or platform systems.
• Proven AI testing automation in production environments.
• Experience building validation and testing frameworks for distributed systems.

Responsibilities

• Build automated validation frameworks and test harnesses across the AI lifecycle.
• Design an AI testing platform integrated with Databricks and MLflow.
• Create large-scale, scenario-based test suites for agentic workflows.
• Validate orchestration behavior and stress non-deterministic systems.
• Embed quality by design with contracts, guardrails, and safe-degradation patterns.
• Define measurable quality signals and integrate into CI/CD gates.
• Ensure validation runs on model, prompt, and code changes for continuous quality.
• Build reusable libraries so teams adopt consistent AI quality practices.

Skills

Backend engineering

AI testing automation

Distributed systems

Python

TypeScript

CI/CD

AWS

Security & risk

Quality gates

Tools

Databricks

MLflow

Job description

Abacus Insights is building AI-powered platforms that transform critical healthcare decisions. The Senior AI Systems Quality Engineer ensures production-ready, trustworthy AI systems by embedding guardrails, evaluation signals, and safe-failure behavior from design to deployment.

You will lead the development of a Databricks‑native AI testing platform and collaborate with AI engineers and delivery teams to define robust evaluation pipelines and quality gates.