Clinical AI Quality Evaluator (Physician)

OpenTrain AI; Northern (KY); Hybrid; USD 124,000 - 207,000; Part time

Job summary

OpenTrain AI is recruiting physicians to evaluate and improve clinical AI systems through structured expert review. You will review model responses, clinical dialogues, reasoning traces, grading criteria, and difficult case questions across clinical streams aligned with your specialty and interests.

This is non-clinical evaluation work with no patient care or live diagnosis. The focus is on accuracy, safety, completeness, appropriate uncertainty, and escalation advice.

Qualifications

• MD or DO with a completed residency in any specialty.
• Active, unrestricted medical license in your country of practice.
• At least two years of post-residency clinical experience in a relevant specialty.
• Ability to evaluate clinical answers for safety, completeness, appropriate uncertainty, and escalation advice.
• Ability to write structured clinical rationale and specialty-specific grading criteria.
• Fluency in written and spoken English.
• Availability for at least 20 hours per week, including time-boxed workstreams.

Responsibilities

• Develop grading criteria that break ideal answers into clear, checkable standards.
• Evaluate multi-turn clinical conversations for accuracy, safety, completeness, appropriate hedging, and escalation guidance.
• Annotate clinical reasoning, including considered and rejected differentials.
• Flag hallucinated findings, dangerous omissions, unsupported certainty, and clinically unsafe advice.
• Define specialty-specific edge cases and standards of care.
• Write difficult clinical questions that probe the limits of model reasoning.

Skills

English fluency

Education

MD or DO with residency

Job description

OpenTrain AI is recruiting physicians to evaluate and improve clinical AI systems through structured expert review. You will review model responses, clinical dialogues, reasoning traces, grading criteria, and difficult case questions across clinical streams aligned with your specialty and interests.

This is non-clinical evaluation work with no patient care or live diagnosis. The focus is on accuracy, safety, completeness, appropriate uncertainty, and escalation advice.