Medical AI Evaluator (MD/DO) – Part-Time, US-Based

Mindrift; Ann Arbor (MI); On-site; USD 55,000 - 138,000; Part time

Job summary

Mindrift invites US-based physicians (MD/DO) to evaluate clinical EHR vignettes for an AI evaluation program focused on testing and improving AI systems. Projects are part-time and error-tolerant, with tasks including vignette scoring and writing short rationales.

Contributors work in blind reads with physician adjudication to ensure high-quality benchmarks. Tasks are around 10-20 hours per week during active phases, and compensation is paid per accepted task, up to $100 per hour.

Qualifications

• Medical degree (MD or DO) and an active, unrestricted US medical license (verified with the issuing medical board).
• Board certification or completed residency training, with recent direct patient care (attending-level preferred).
• Broad, multi-system diagnostic and treatment experience (internal, family, emergency, or hospital medicine).
• Strong clinical reasoning: differential diagnosis, next-step management, and application of evidence-based guidelines.
• Prior experience in medical AI evaluation, clinical content or exam-question review, or medical annotation/QA (a plus).
• Strong written English (C1+).

Responsibilities

• Evaluate clinical EHR vignettes paired with questions, proposed answers, and distractors across multiple systems.
• Score reasoning quality: vignette accuracy, completeness, and trap quality.
• Check whether reasoning chain follows vignette facts and write short rationales.
• Participate in three blind reads followed by physician adjudication.
• Contribute to improving AI tools used clinically.

Skills

Active US medical license

Strong clinical reasoning

Broad multi-system diagnostic exp

Strong written English (C1+)

Experience in medical AI evaluation/QA

Education

MD or DO

Board certification or residency training

Job description

Mindrift invites US-based physicians (MD/DO) to evaluate clinical EHR vignettes for an AI evaluation program focused on testing and improving AI systems. Projects are part-time and error-tolerant, with tasks including vignette scoring and writing short rationales.

Contributors work in blind reads with physician adjudication to ensure high-quality benchmarks. Tasks are around 10-20 hours per week during active phases, and compensation is paid per accepted task, up to $100 per hour.