Senior AI Agent Reliability Engineer

Claritev Corporation

United States

On-site

USD 140,000 - 160,000

Full time

8 days ago

Application generator

Don’t send a generic resume — generate a resume and cover letter tailored to this exact role.

Get past ATS filters

Benefits offered by this job

Health insurance

401(k) with company match

Bonus opportunity

Job summary

Claritev seeks a Senior AI Operations Engineer to gatekeep agentic AI systems powering healthcare products. You will design and run tests, benchmarks, and evaluation pipelines to catch regressions, hallucinations, and unsafe actions before release.

Collaborate with AI/ML teams and product stakeholders to build golden datasets, automate tests in CI/CD, triage agent failures, and translate results into quality insights while ensuring PHI/PII handling and HIPAA compliance.

Qualifications

Bachelor's degree in CS, engineering, data science, or related field.
5+ years in software QA/test automation or ML engineering.
2+ years with production ML/AI systems.
1+ years with generative AI, LLMs, or agentic AI.
Track record building automation improving software/model quality.

Responsibilities

Design, build, and maintain automated test suites and evaluation pipelines for agentic AI systems.
Develop golden datasets and simulation environments reflecting healthcare workflows.
Automate agent benchmarking and CI/CD quality gates.
Measure and report agent quality metrics and safety compliance.
Implement LLM-as-judge and rubric-based scoring; verify with human review.
Triage, reproduce, and root-cause agent failures; file defect reports.
Perform adversarial testing and guardrail verification.
Monitor production agent behavior and feed findings into offline tests.
Contribute to evaluation dashboards, runbooks, and quality docs.
Ensure PHI/PII handling and HIPAA compliance; support data governance.