Remote Medical AI Evaluation Specialist

YO AI Labs; Seattle (WA); Remote; USD 55,000 - 96,000; Full time

Job summary

YO AI Labs seeks Medical Evaluation Specialists, including medical students, residents, physicians, and biomedical professionals, to contribute clinical expertise to evaluating and improving next‑generation AI systems.

In this contractor role, you will create and validate challenging medical questions and answers to test AI systems' ability to interpret evidence, synthesize information, and apply clinical judgment.

Qualifications

• Medical training and clinical knowledge.
• Ability to locate and appraise primary literature.
• Strong written English with precise documentation.
• Experience in medical education or content development is a plus.

Responsibilities

• Create original, high-difficulty medical question-and-answer pairs covering areas such as diagnosis, pathophysiology, pharmacology, clinical guidelines, and clinical decision-making.
• Develop questions that require synthesis, interpretation, and genuine medical reasoning rather than simple factual recall.
• Research and verify answers using primary literature, clinical guidelines, systematic reviews, and authoritative medical references.
• Document the rationale behind answers and provide appropriate supporting citations.
• Evaluate AI-generated responses for clinical accuracy, completeness, reasoning quality, and adherence to available evidence.
• Identify questions that are too straightforward and refine them to increase complexity while maintaining clinical validity.
• Review questions and answers for ambiguity, unsupported assumptions, factual errors, and inconsistencies.
• Ensure all deliverables are written with clarity, precision, and defensibility.
• Incorporate reviewer feedback and adapt work to evolving project guidelines and quality standards.
• Contribute insights that help establish rigorous benchmarks for medical AI evaluation.

Skills

Medical Training

Research & Source Triangulation

Attention to Detail

Written Precision

Analytical Thinking

Clinical Reasoning

Medical Literature Review

Evidence Synthesis

Self-Direction & Reliability

Medical Question Development

Education

Medical student, resident, or physician

Job description

YO AI Labs seeks Medical Evaluation Specialists, including medical students, residents, physicians, and biomedical professionals, to contribute clinical expertise to evaluating and improving next‑generation AI systems.

In this contractor role, you will create and validate challenging medical questions and answers to test AI systems' ability to interpret evidence, synthesize information, and apply clinical judgment.