Psychiatry AI Evaluation & Case Design Specialist

mpathic; United States; Remote; USD 110,000 - 165,000; Part time

Job summary

mpathic.ai is seeking experienced psychiatrists to help evaluate and improve the next generation of large language models by testing how they reason through complex psychiatric cases. You will create challenging, clinically realistic scenarios that push AI beyond straightforward textbook knowledge.

You'll identify where models break down, develop expert reference answers grounded in evidence-based psychiatric practice, and provide structured feedback that directly improves model

Qualifications

• MD, DO, or internationally equivalent medical degree.
• Completion of an accredited psychiatry residency.
• Current board certification in Psychiatry (ABPN, FRCPC, MRCPsych, FRANZCP, or recognized national equivalent).
• Current, active, unrestricted medical license with no suspension, revocation, or ongoing disciplinary action.
• At least 3 years of independent clinical practice following residency (training years excluded).
• Experience practicing in an accredited or government-licensed healthcare setting, including hospitals, academic medical centers, community mental health clinics, private practice, or telepsychiatry.

Responsibilities

• Design clinically realistic prompts and case scenarios based on your psychiatry practice.
• Evaluate AI-generated responses using structured rubrics for accuracy, safety, completeness, and clinical judgment.
• Identify nuanced edge cases, challenging differential diagnoses, and common model failure modes to improve AI reasoning.
• Collaborate with researchers at a leading AI lab to advance healthcare AI systems.
• Provide detailed written feedback that informs future model training and evaluation.
• Participate in onboarding, calibration sessions, and ongoing quality reviews.

Skills

Direct patient care

OSCE experience

Telepsychiatry

Education

MD/DO or international equivalent medical degree

ACGME/AOA psychiatry residency

Board certification in Psychiatry

Active, unrestricted medical license

Job description

mpathic.ai is seeking experienced psychiatrists to help evaluate and improve the next generation of large language models by testing how they reason through complex psychiatric cases. You will create challenging, clinically realistic scenarios that push AI beyond straightforward textbook knowledge.

You'll identify where models break down, develop expert reference answers grounded in evidence-based psychiatric practice, and provide structured feedback that directly improves model