Senior AI Quality Engineer for Production Test Automation

Abacus Insights; Northern (KY); Hybrid; USD 140,000 - 190,000; Full time

Job summary

Abacus Insights is building AI‑powered platforms focused on reliable and trustworthy healthcare data. As Senior AI Systems Quality Engineer, you will embed guardrails and testing into agentic AI systems from the ground up, influencing release readiness and governance.

This role emphasizes automation‑first practices, platform ownership, and scalable quality across complex distributed systems in regulated environments. Collaboration with AI engineers and platform teams is essential.

Qualifications

• Experience designing and implementing AI testing in production environments.
• Ability to build custom validation and testing frameworks for distributed systems.
• Strong Python and/or TypeScript skills in modern AI stacks.
• Knowledge of CI/CD integration and automated quality gates.

Responsibilities

• Build production‑grade automated validation frameworks and evaluation pipelines across the AI lifecycle.
• Design and evolve an AI testing platform integrated with Databricks and MLflow.
• Create large‑scale, scenario‑based test suites to validate agentic workflows end‑to‑end.

Skills

Backend systems

AI testing automation

Python

TypeScript

CI/CD

AWS

Security & risk

Education

Bachelor's degree in CS or related

Tools

Databricks

MLflow

Prometheus

Job description

Abacus Insights is building AI‑powered platforms focused on reliable and trustworthy healthcare data. As Senior AI Systems Quality Engineer, you will embed guardrails and testing into agentic AI systems from the ground up, influencing release readiness and governance.

This role emphasizes automation‑first practices, platform ownership, and scalable quality across complex distributed systems in regulated environments. Collaboration with AI engineers and platform teams is essential.