Senior AI Reliability Engineer (Platform QA)

Jobs in JS; Boston (MA); On-site; USD 140,000 - 190,000; Full time

Benefits offered by this job

Unlimited PTO

Work from anywhere

Health coverage

Equity

Growth‑focused environment

Home office setup allowance

Cell phone allowance

Job summary

Abacus Insights in Boston, MA is seeking a Senior AI Systems Quality Engineer to embed quality into agentic AI platforms that support critical healthcare decisions. You will help design guardrails, evaluation signals, and safe‑degradation patterns from the ground up, ensuring safety, reliability and trust in production systems.

This role is platform‑oriented: build end‑to‑end automated AI testing integrated into the development lifecycle, design a Databricks/MLflow powered testing platform,

Qualifications

• 7+ years of software engineering experience, backend or platform systems.
• Experience designing AI testing automation in production environments.
• Ability to build custom validation, evaluation, or testing frameworks for distributed systems.
• Strong Python and/or TypeScript proficiency.
• Experience with AI testing at scale, including regression frameworks and large test coverage.
• Understanding CI/CD with automated tests and quality gates.
• Knowledge of AWS cloud-native architectures.
• Security, privacy, and operational risk in regulated environments.
• Experience with non-deterministic outputs evaluation and guardrails.

Responsibilities

• Build production‑grade automated validation frameworks, test harnesses, and evaluation pipelines across the AI lifecycle.
• Design and evolve an AI testing platform integrated with Databricks and MLflow for repeatable testing and auditability.
• Create large‑scale scenario-based test suites to validate agentic workflows end‑to‑end.
• Validate orchestration behavior and stress‑test non‑deterministic system behavior before production.
• Embed quality by design with system contracts, guardrails, and safe‑degradation patterns.
• Define measurable quality signals for LLM systems and integrate them into CI/CD pipelines as automated gates.
• Ensure AI validation runs automatically on model, prompt, and code changes for continuous quality.
• Build reusable libraries and components to standardize AI quality practices.
• Own AI release readiness with go/no-go criteria based on quality thresholds.

Skills

Python

TypeScript

AI testing

CI/CD

AWS

Reliability

Security

Observability

Non-determinism

Quality gates

Tools

Databricks

MLflow

Job description

Abacus Insights in Boston, MA is seeking a Senior AI Systems Quality Engineer to embed quality into agentic AI platforms that support critical healthcare decisions. You will help design guardrails, evaluation signals, and safe‑degradation patterns from the ground up, ensuring safety, reliability and trust in production systems.

This role is platform‑oriented: build end‑to‑end automated AI testing integrated into the development lifecycle, design a Databricks/MLflow powered testing platform,