Remote Medical AI Evaluation Architect

YO AI Labs; Philadelphia (Philadelphia County); Remote; USD 60,000 - 110,000; Part time

Job summary

YO AI Labs in the United States seeks Medical Evaluation Specialists (medical students, residents, physicians, and biomedical professionals) to create and validate challenging medical questions and answers for AI evaluation.

You will research primary literature and guidelines, document rationales with citations, evaluate AI responses for accuracy and reasoning, and refine questions to increase clinical difficulty while maintaining validity.

Qualifications

• Medical training required.
• Strong written English skills with exceptional attention to detail.
• Ability to locate, interpret, and synthesize medical literature from primary sources and guidelines.
• Analytical thinking and clinical reasoning for complex medical scenarios.
• Experience with evidence-based medicine and literature review.

Responsibilities

• Create original, high-difficulty medical question-and-answer pairs across diagnosis, pathophysiology, pharmacology, guidelines, and clinical decision-making.
• Develop questions that require synthesis, interpretation, and clinical reasoning rather than recall.
• Research and verify answers using primary literature, clinical guidelines, systematic reviews, and authoritative references.
• Document the rationale behind answers with proper supporting citations.
• Evaluate AI-generated responses for clinical accuracy, completeness, and reasoning quality.
• Refine questions that are too straightforward to increase complexity while preserving validity.
• Review questions for ambiguity, unsupported assumptions, and inconsistencies.
• Ensure all deliverables are clear, precise, and defensible.
• Incorporate reviewer feedback and adapt to evolving project guidelines.

Skills

Medical Training

Research & Source Triangulation

Attention to Detail

Written Precision

Analytical Thinking

Clinical Reasoning

Medical Literature Review

Evidence Synthesis

Self-Direction & Reliability

Medical Question Development

Education

Medical student / resident / physician training

Job description

YO AI Labs in the United States seeks Medical Evaluation Specialists (medical students, residents, physicians, and biomedical professionals) to create and validate challenging medical questions and answers for AI evaluation.

You will research primary literature and guidelines, document rationales with citations, evaluate AI responses for accuracy and reasoning, and refine questions to increase clinical difficulty while maintaining validity.