Physician AI Evaluation Specialist (MD/DO) - Part-Time

Mindrift; Pittsburgh (Allegheny County); Remote; USD 114,000 - 162,000; Part time

Job summary

Mindrift connects physicians with project-based AI evaluation opportunities for leading tech companies. This role focuses on testing and improving AI medical systems, based in the United States.

We seek US-based physicians (MD/DO) to evaluate EHR vignettes and score clinical reasoning quality across multiple domains. Expect participation in three blind reads with physician adjudication, on a part-time basis.

Qualifications

• MD or DO with active US medical license (verified with the issuing medical board).
• Board certification or completed residency with recent direct patient care (attending-level preferred).
• Broad, multi-system diagnostic and treatment experience (internal, family, emergency, or hospital medicine).
• Strong clinical reasoning: differential diagnosis, next-step management, and evidence-based guidelines.

Responsibilities

• Evaluate clinical EHR vignettes paired with a question, a proposed answer, and a distractor answer across multiple medical domains.
• Score the clinical reasoning quality of benchmark items and assess vignette accuracy, completeness, and trap quality.
• Check whether the reasoning chain reaches the answer from vignette facts alone and write short rationales.
• Participate in three independent blind reads followed by physician adjudication.

Skills

Clinical reasoning

Medical knowledge

Excellent written English

Education

MD or DO

Active US medical license

Job description

Mindrift connects physicians with project-based AI evaluation opportunities for leading tech companies. This role focuses on testing and improving AI medical systems, based in the United States.

We seek US-based physicians (MD/DO) to evaluate EHR vignettes and score clinical reasoning quality across multiple domains. Expect participation in three blind reads with physician adjudication, on a part-time basis.