Part-Time Physician AI Evaluation Specialist (MD/DO)

Mindrift; North Carolina; Remote; USD 114,000 - 162,000; Full time

Job summary

Mindrift seeks US-based, actively practicing physicians (MD/DO) to evaluate clinical EHR vignettes for a medical AI evaluation program. General and internal medicine are the main focus. Projects are task-based and may involve varying duties across multiple systems.

Contributors will evaluate vignettes, score reasoning quality, check reasoning traces, and write concise rationales. The process uses three independent blind reads with physician adjudication to ensure accuracy.

Qualifications

• MD or DO with active, unrestricted US medical license.
• Board certification or completed residency with recent direct patient care.
• Broad multi-system diagnostic and treatment experience (internal, family, emergency, or hospital medicine).
• Strong clinical reasoning and application of evidence-based guidelines.
• Prior experience in medical AI evaluation or medical annotation/QA is a plus.
• Excellent written English (C1+).

Responsibilities

• Evaluate clinical EHR vignettes paired with a question, a proposed answer, and a distractor.
• Score the clinical reasoning quality of benchmark items: vignette accuracy, answerability, and trap quality.
• Check whether the reasoning chain reaches the answer from vignette facts and write short rationales.
• Participate in three blind reads followed by physician adjudication.

Job description

Mindrift seeks US-based, actively practicing physicians (MD/DO) to evaluate clinical EHR vignettes for a medical AI evaluation program. General and internal medicine are the main focus. Projects are task-based and may involve varying duties across multiple systems.

Contributors will evaluate vignettes, score reasoning quality, check reasoning traces, and write concise rationales. The process uses three independent blind reads with physician adjudication to ensure accuracy.