Clinical AI Evaluation Physician

OpenTrain AI; Northern (KY); Hybrid; USD 124,000 - 207,000; Part time

Job summary

OpenTrain AI is recruiting physicians to evaluate and improve clinical AI systems through structured expert review. You will review model responses, clinical dialogues, reasoning traces, grading criteria, and difficult case questions across clinical streams aligned with your specialty and interests.

This is non-clinical evaluation work with no patient care or live diagnosis. The focus is on accuracy, safety, completeness, appropriate uncertainty, and escalation advice.

Qualifications

• MD or DO with a completed residency in any specialty.
• Active, unrestricted medical license in your country of practice.
• At least two years of post-residency clinical experience in a relevant specialty.
• Ability to evaluate clinical answers for safety, completeness, appropriate uncertainty, and escalation advice.
• Ability to write structured clinical rationale and specialty-specific grading criteria.
• Fluency in written and spoken English.
• Availability for at least 20 hours per week, including time-boxed workstreams.

Responsibilities

• Develop grading criteria that break ideal answers into clear, checkable standards.
• Evaluate multi-turn clinical conversations for accuracy, safety, completeness, appropriate hedging, and escalation guidance.
• Annotate clinical reasoning, including considered and rejected differentials.
• Flag hallucinated findings, dangerous omissions, unsupported certainty, and clinically unsafe advice.
• Define specialty-specific edge cases and standards of care.
• Write difficult clinical questions that probe the limits of model reasoning.

Skills

English fluency

Education

MD or DO with residency

Job description

About OpenTrain

OpenTrain AI is the hiring and contracting organization for this opportunity. OpenTrain is the #1 platform for finding and building careers in AI training and data labeling, helping contributors discover projects, build a professional profile, and grow their experience in a rapidly developing field.

• Create a free OpenTrain account to apply.
• Build a profile that reflects your clinical and AI-training experience.
• Explore opportunities that align with your skills and interests.

About Clinical AI Training

AI training is the human work behind modern artificial intelligence. Expert reviewers assess model outputs, identify errors, and create high-quality examples and standards that help AI systems become more accurate, safe, useful, and appropriately uncertain.

In this role, your clinical expertise will help measure whether AI systems provide sound reasoning, complete answers, responsible escalation guidance, and advice that is appropriate for the information available.

• Work on cutting-edge systems without providing patient care.
• Use clinical judgment to improve how AI handles difficult cases.
• Contribute to a growing field that supports flexible, technology-focused work.

The Role

OpenTrain is recruiting physicians to evaluate and improve clinical AI systems through structured expert review. You will review model responses, clinical dialogues, reasoning traces, grading criteria, and difficult case questions across clinical streams aligned with your specialty and interests.

This is non-clinical evaluation work. It does not involve patient care or live diagnosis. The focus is on measuring accuracy, safety, completeness, appropriate uncertainty, and escalation advice.

• Role: Clinical AI Evaluation Physician
• Engagement: Part-time contractor
• Location eligibility: United States
• Language: English
• Time commitment: At least 20 hours per week
• Pay: $150 USD per hour

What You’ll Do

You will turn expert clinical judgment into clear evaluation standards and apply those standards consistently when reviewing AI-generated content. Your work will help identify both obvious errors and subtle risks in clinical reasoning and communication.

• Develop grading criteria that break ideal answers into clear, checkable standards.
• Evaluate multi-turn clinical conversations for accuracy, safety, completeness, appropriate hedging, and escalation guidance.
• Annotate clinical reasoning, including considered and rejected differentials.
• Flag hallucinated findings, dangerous omissions, unsupported certainty, and clinically unsafe advice.
• Define specialty-specific edge cases and standards of care.
• Write difficult clinical questions that probe the limits of model reasoning.

Required Qualifications

This opportunity requires substantial clinical training and experience, along with the ability to communicate medical reasoning clearly to both specialists and non-specialist reviewers.

• MD or DO with a completed residency in any specialty.
• Active, unrestricted medical license in your country of practice.
• At least two years of post-residency clinical experience in a relevant specialty.
• Ability to evaluate clinical answers for safety, completeness, appropriate uncertainty, and escalation advice.
• Ability to write structured clinical rationale and specialty-specific grading criteria.
• Fluency in written and spoken English.
• Availability for at least 20 hours per week, including concentrated time when a workstream is time-boxed.

Helpful Background

The following experience is valuable but is not listed as required. It may help you contribute effectively to specialty-specific evaluation standards and challenging clinical cases.

• Board certification.
• U.S. medical licensure or familiarity with U.S. standards of care.
• Experience in primary care, internal medicine, emergency medicine, or hospital medicine.
• Clinical annotation or AI evaluation.
• Medical education or question writing.
• Grading-criteria design or guideline development.
• Clinical research or technical writing.

Why Join OpenTrain

AI training gives subject-matter experts a way to apply their professional knowledge to the development of advanced technology. OpenTrain helps you manage opportunities in one place, showcase credible experience, and build a longer-term portfolio in AI training and data labeling.

• Apply your medical expertise to meaningful AI evaluation work.
• Develop experience at the intersection of clinical practice and artificial intelligence.
• Build a profile that can support future AI-training opportunities.
• Work as an independent contractor with part-time availability.