Part-Time Medical AI Evaluator (MD/DO)

Braintrust; Bismarck (ND); Remote; USD 138,000 - 207,000; Part time

Job summary

Mindrift connects physicians with project-based AI opportunities for evaluating and improving medical AI tools. We are seeking US-based physicians (MD/DO) to participate in evaluating clinical EHR vignettes for a Medical AI Evaluation Program, focusing on general and internal medicine.

This is a part-time, project-based role, not permanent employment. Responsibilities include evaluating vignette items, scoring reasoning quality, verifying reasoning traces, and writing concise rationales.

Qualifications

• MD or DO with active US medical license.
• Board certification or completed residency training.
• Broad multi-system diagnostic and treatment experience.

Responsibilities

• Evaluate clinical EHR vignettes with questions and distractors across multiple systems.
• Score clinical reasoning quality: vignette accuracy, completeness, and trap quality.
• Assess whether reasoning chains reach conclusions from vignette facts and write brief rationales.
• Contribute to three independent blind reads and physician adjudication.

Skills

Clinical reasoning

Multisystem medicine

AI evaluation experience

Written English

Education

MD/DO

Active US medical license

Board certification or residency

Job description

Mindrift connects physicians with project-based AI opportunities for evaluating and improving medical AI tools. We are seeking US-based physicians (MD/DO) to participate in evaluating clinical EHR vignettes for a Medical AI Evaluation Program, focusing on general and internal medicine.

This is a part-time, project-based role, not permanent employment. Responsibilities include evaluating vignette items, scoring reasoning quality, verifying reasoning traces, and writing concise rationales.