Medical AI Evaluator (MD/DO) — EHR Vignettes, Part-Time

Mindrift; Eagle (ID); On-site; USD 83,000 - 179,000; Part time

Job summary

Mindrift connects specialists with project-based AI opportunities for leading tech companies, focused on testing, evaluating, and improving AI systems. Participation is project-based.

We’re looking for US-based, actively practicing physicians to evaluate clinical EHR vignettes for a medical AI evaluation program. Tasks include scoring reasoning quality, ensuring explanations are clear, and writing rationales for three blind reads with adjudication.

Qualifications

• Medical degree (MD or DO) and active US medical license.
• Board certification or residency training with recent direct patient care.
• Broad, multi-system diagnostic and treatment experience.
• Strong clinical reasoning and knowledge of guidelines.
• Experience in medical AI evaluation or QA is a plus.
• Strong written English (C1+).

Responsibilities

• Evaluate clinical EHR vignettes with questions, proposed answers, and distractors.
• Score the clinical reasoning quality of vignette items and distractor quality.
• Check whether the reasoning chain reaches the answer from vignette facts; write short rationales.
• Work within three independent blind reads, followed by physician adjudication.
• Contribute to improving AI tools through evaluation feedback.

Skills

Active US medical license

Clinical reasoning

Medical AI evaluation experience

Strong written English

Vignette evaluation

Education

MD/DO degree

US medical license (board-verified)

Board certification or residency training

Job description

Mindrift connects specialists with project-based AI opportunities for leading tech companies, focused on testing, evaluating, and improving AI systems. Participation is project-based.

We’re looking for US-based, actively practicing physicians to evaluate clinical EHR vignettes for a medical AI evaluation program. Tasks include scoring reasoning quality, ensuring explanations are clear, and writing rationales for three blind reads with adjudication.