Lead AI Systems Quality Engineer - Automation & Platform

F5 Remote Jobs; Northern (KY); Hybrid; USD 150,000 - 210,000; Full time

Benefits offered by this job

Unlimited PTO

Work from anywhere

Comprehensive health coverage

Equity for all employees

Growth-focused environment

Home office setup allowance

Phone allowance

Job summary

Abacus Insights is building AI-driven platforms for healthcare decisions, focusing on reliability, correctness, and safety as we scale agentic systems. The Senior AI Systems Quality Engineer will help design end-to-end automated testing, guardrails, and a Databricks-MLflow-based evaluation stack to ensure production readiness.

You will collaborate with AI engineers, platform and security teams to define quality criteria and release readiness, shaping how scalable, governable AI is deployed in

Qualifications

• 7+ years of software engineering experience, backend or platform-focused.
• Proven AI testing automation experience in production environments.
• Experience building validation or testing frameworks for distributed systems.
• Strong Python and/or TypeScript skills.

Responsibilities

• Build automated AI validation frameworks and evaluation pipelines across the AI lifecycle.
• Design an AI testing platform integrated with Databricks and MLflow for traceability and auditability.
• Create large-scale test suites covering edge and long-tail scenarios.
• Validate orchestration behavior and non-deterministic system performance before production.
• Embed quality by design with contracts, guardrails, and safe-degradation patterns.
• Define measurable quality signals for LLM systems and integrate into CI/CD gates.
• Ensure validation runs on model, prompt, and code changes for continuous quality.
• Develop reusable libraries to standardize AI quality practices across teams.
• Define go/no-go criteria for release readiness with stakeholders.

Job description

Abacus Insights is building AI-driven platforms for healthcare decisions, focusing on reliability, correctness, and safety as we scale agentic systems. The Senior AI Systems Quality Engineer will help design end-to-end automated testing, guardrails, and a Databricks-MLflow-based evaluation stack to ensure production readiness.

You will collaborate with AI engineers, platform and security teams to define quality criteria and release readiness, shaping how scalable, governable AI is deployed in