Healthcare AI Systems Evaluator

Empathy; United States; On-site; USD 82,656 - 165,312; Full time

Job summary

mpathic.ai is seeking experienced healthcare systems professionals to evaluate and improve AI models that support clinical operations, administrative workflows, and healthcare technology. You will review real-world scenarios, evaluate AI-generated responses, and provide expert feedback to improve reliability for providers, staff, and patients.

Bring deep knowledge of healthcare workflows across hospitals and health systems, familiarity with EHR platforms, and the ability to communicate complex

Qualifications

• 5+ years in healthcare operations or related healthcare systems role.
• Strong understanding of hospital and health system workflows.
• Experience with EHR platforms like Epic, Cerner/Oracle Health, Meditech, athenahealth.
• Familiarity with HL7/FHIR CCD/C-CDA data exchange.
• Knowledge of HIPAA and CMS requirements.
• Strong written English and clear communication.

Responsibilities

• Evaluate AI-generated responses for accuracy and alignment with real-world operations.
• Review scenarios involving electronic health records (EHRs), clinical documentation, care coordination, scheduling, referrals, revenue cycle, and patient access workflows.
• Assess AI outputs for consistency with healthcare regulations, industry standards, and operational best practices.
• Apply structured evaluation guidelines to annotate model responses and identify errors, omissions, and opportunities for improvement.
• Provide clear, actionable feedback that helps improve AI performance across complex healthcare workflows.
• Evaluate AI handling of interoperability, health information exchange, and healthcare data management concepts.

Skills

Healthcare ops

EHR workflows

Regulatory knowledge

English writing

Tools

Epic

Cerner

Meditech

athenahealth

Job description

mpathic.ai is seeking experienced healthcare systems professionals to evaluate and improve AI models that support clinical operations, administrative workflows, and healthcare technology. You will review real-world scenarios, evaluate AI-generated responses, and provide expert feedback to improve reliability for providers, staff, and patients.

Bring deep knowledge of healthcare workflows across hospitals and health systems, familiarity with EHR platforms, and the ability to communicate complex