Remote Medical Evaluation Specialist for AI Benchmarking

YO AI Labs; Boston (MA); Remote; USD 55,000 - 110,000; Full time

Job summary

YO AI Labs is seeking Medical Evaluation Specialists, including medical students, residents, physicians, and biomedical professionals, to contribute clinical expertise to a project evaluating next-generation AI systems.

You will create and validate high-difficulty medical questions and answers to test advanced clinical reasoning, and document rationales with citations. No prior AI experience required; strong medical knowledge and research skills matter most.

Qualifications

• Medical training and clinical expertise relevant to evaluating AI medical content.
• Ability to locate and interpret primary literature and guidelines.
• Strong written English with precise, defensible reasoning.
• Experience in creating high-quality medical questions and answers.

Responsibilities

• Create original, high-difficulty medical Q&A pairs across diagnosis, pathophysiology, pharmacology, and guidelines.
• Research and verify answers using primary literature and authoritative references.
• Document rationale with supporting citations for each answer.
• Evaluate AI-generated responses for clinical accuracy and reasoning.
• Refine questions to increase complexity while preserving validity.
• Review for ambiguity and inconsistencies; maintain clarity and defensibility.
• Incorporate reviewer feedback and adapt to project guidelines.

Skills

Medical Training

Research & Source Triangulation

Attention to Detail

Written Precision

Analytical Thinking

Clinical Reasoning

Medical Literature Review

Evidence Synthesis

Self-Direction & Reliability

Medical Question Development

Job description

YO AI Labs is seeking Medical Evaluation Specialists, including medical students, residents, physicians, and biomedical professionals, to contribute clinical expertise to a project evaluating next-generation AI systems.

You will create and validate high-difficulty medical questions and answers to test advanced clinical reasoning, and document rationales with citations. No prior AI experience required; strong medical knowledge and research skills matter most.