Senior AI Agent QA Engineer for CI/CD & MLOps

MultiPlan; McLean (VA); On-site; USD 140,000 - 160,000; Full time

Benefits offered by this job

Health insurance

401k

Bonus opportunity

Job summary

Claritev in McLean, Virginia, is seeking a Senior AI Operations Engineer to serve as the quality gatekeeper for agentic AI systems powering our healthcare products. You will design and run test suites, benchmarks, and evaluation pipelines to validate agent behavior before and after release.

Collaborate with AI engineers, data scientists, and product teams to build golden datasets, automate testing in CI/CD, triage and reproduce failures, and translate evaluation results into actionable quality

Qualifications

• Bachelor's degree in Computer Science, Engineering, Data Science, or a related quantitative field.
• Master's degree is a plus.
• 5+ years hands-on experience in software engineering, QA/test automation, ML engineering, or related technical discipline.
• 2+ years testing/operating production ML/AI systems.
• 1+ years hands-on with generative AI, LLMs, RAG, or agentic AI systems.
• Track record building test automation or evaluation capabilities.

Responsibilities

• Design, build, and maintain automated test suites and evaluation pipelines for agentic AI systems, covering single-turn, multi-turn, and end-to-end workflow scenarios.
• Develop golden datasets, test scenarios, and simulation environments reflecting real healthcare workflows and edge cases.
• Execute and automate agent benchmarking, including regression testing across model, prompt, and tool changes, with clear pass/fail quality gates in CI/CD.
• Measure and report agent quality metrics: task completion, tool-call accuracy, grounding/hallucination rates, response quality, latency, cost per task, and safety compliance.
• Implement LLM-as-judge and rubric-based scoring approaches, and validate automated scores against human review.
• Triage, reproduce, and root-cause agent failures by analyzing traces of LLM calls, tool invocations, retrieval steps, and orchestration paths; file actionable defect reports.
• Perform adversarial and negative testing, including prompt-injection probes, malformed inputs, boundary conditions, and guardrail verification.
• Monitor production agent behavior, investigate quality incidents and drift, and feed findings back into offline test coverage.
• Contribute to evaluation dashboards, runbooks, and quality documentation used by engineering and product teams.
• Verify PHI/PII handling, auditability, and human-in-the-loop controls behave as designed, supporting HIPAA and data-governance compliance.
• Collaborate with AI engineers and data scientists to make agents more testable, observable, and reliable by design.

Skills

Python

pytest

APIs

distributed systems

LLM evaluation

LangChain

Kubernetes

CI/CD

observability tooling

statistics

guardrails

Education

Bachelor's degree in Computer Science, Engineering, Data Science, a quantitative discipline, or a related field

Master's degree a plus

Tools

LangSmith

Langfuse

Arize Phoenix

Ragas

DeepEval

promptfoo

LangGraph

LangChain

AutoGen

CrewAI

Job description

Claritev in McLean, Virginia, is seeking a Senior AI Operations Engineer to serve as the quality gatekeeper for agentic AI systems powering our healthcare products. You will design and run test suites, benchmarks, and evaluation pipelines to validate agent behavior before and after release.

Collaborate with AI engineers, data scientists, and product teams to build golden datasets, automate testing in CI/CD, triage and reproduce failures, and translate evaluation results into actionable quality