Part-Time US Physician: AI Clinical Vignette Evaluator

Braintrust; Pierre (SD); Remote; USD 83,000 - 165,000; Part time

Job summary

Braintrust seeks US-based, actively practicing physicians (MD/DO) to evaluate clinical EHR vignettes for a Medical AI evaluation program. This is project-based, with part-time hours, and focuses on General and Internal Medicine.

Contributors will score reasoning quality, assess vignette accuracy, provide brief rationales, and work through blind reads with physician adjudication. Candidates must hold a medical degree, an active US license, and strong English writing skills.

Qualifications

• Medical degree (MD or DO) and an active, unrestricted US medical license.
• Board certification or completed residency training with recent direct patient care.
• Broad, multi-system diagnostic and treatment experience (internal, family, emergency, or hospital medicine).
• Strong clinical reasoning: differential diagnosis, next-step management, and evidence-based guidelines.
• Prior experience in medical AI evaluation, clinical content or exam-question review, or medical annotation/QA.
• Strong written English (C1+).

Responsibilities

• Evaluate clinical EHR vignettes paired with a question, a proposed answer, and a distractor “trap” answer across diagnosis and treatment tasks.
• Score the clinical reasoning quality of benchmark items: vignette accuracy and completeness, whether the vignette gives the answer away, answer correctness and gradeability, and trap quality.
• Check whether the reasoning chain reaches the answer from vignette facts alone, correct reasoning traces, and write short rationales.
• Work within three independent blind reads, followed by physician adjudication.
• Contribute to improving AI tools used in clinical practice.

Skills

Clinical reasoning

Multi-system knowledge

Medical AI evaluation

Strong written English

Education

MD/DO

Active US medical license

Job description

Braintrust seeks US-based, actively practicing physicians (MD/DO) to evaluate clinical EHR vignettes for a Medical AI evaluation program. This is project-based, with part-time hours, and focuses on General and Internal Medicine.

Contributors will score reasoning quality, assess vignette accuracy, provide brief rationales, and work through blind reads with physician adjudication. Candidates must hold a medical degree, an active US license, and strong English writing skills.