Part-Time Medical AI Evaluator for EHR Vignettes (MD/DO)

Braintrust; United States; Remote; USD 110,000 - 179,000; Full time

Job summary

Toloka AI B.V. is seeking US-based physicians (MD/DO) to evaluate clinical EHR vignettes for a medical AI evaluation program. You will assess diagnosis and treatment tasks across multiple systems, scoring reasoning quality and draft concise rationales.

Contributors work on a part-time, project-based basis with three blind reads followed by physician adjudication. A strong English proficiency is required and prior AI evaluation experience is a plus.

Qualifications

• MD or DO degree required and active US medical license.
• Board certification or completed residency training with recent direct patient care.
• Strong clinical reasoning and ability to apply evidence-based guidelines.

Responsibilities

• Evaluate clinical EHR vignettes for an AI evaluation program.
• Score clinical reasoning quality across vignette items and distractors.
• Write concise rationales and ensure traceability of the reasoning.
• Work through independent reads and physician adjudication.

Skills

Medical license

Board certification

Clinical reasoning

Written English

Education

MD or DO degree

Board certification or residency training

Job description

Toloka AI B.V. is seeking US-based physicians (MD/DO) to evaluate clinical EHR vignettes for a medical AI evaluation program. You will assess diagnosis and treatment tasks across multiple systems, scoring reasoning quality and draft concise rationales.

Contributors work on a part-time, project-based basis with three blind reads followed by physician adjudication. A strong English proficiency is required and prior AI evaluation experience is a plus.