Medical AI Evaluation Specialist (MD/DO) — Part-Time

Mindrift; Elm (PA); Remote; USD 48,000 - 96,000; Part time

Job summary

Mindrift connects physicians with project-based AI evaluation roles, focusing on testing and improving AI systems through clinical EHR vignette assessments. US-based physicians will evaluate and score diagnostic vignettes, write concise rationales, and participate in blind reads with adjudication.

This is a part-time, project-based opportunity. Ideal candidates hold an MD or DO with an active license, board certification or recent residency, and strong written English.

Qualifications

• MD or DO with an active, unrestricted US medical license.
• Board certification or completed residency with recent direct patient care.
• Broad, multi-system diagnostic and treatment experience (internal, family, emergency, or hospital medicine).
• Strong clinical reasoning: differential diagnosis, next-step management, and evidence-based guidelines.

Responsibilities

• Evaluate clinical EHR vignettes with questions, proposed answers, and distractors across multiple medical domains.
• Score clinical reasoning quality, vignette accuracy, answer correctness, and trap quality.
• Check whether the reasoning chain reaches the correct conclusion and write short rationales.
• Participate in three independent blind reads followed by physician adjudication to improve AI evaluation tools.

Skills

Strong written English

Education

MD/DO with active US license

Board certification or recent residency

Job description

Mindrift connects physicians with project-based AI evaluation roles, focusing on testing and improving AI systems through clinical EHR vignette assessments. US-based physicians will evaluate and score diagnostic vignettes, write concise rationales, and participate in blind reads with adjudication.

This is a part-time, project-based opportunity. Ideal candidates hold an MD or DO with an active license, board certification or recent residency, and strong written English.