Remote Medical AI Question Architect

YO AI Labs

Austin (TX)

Remote

USD 55,000 - 110,000

Full time

14 days+

Application generator

Turn this role into an interview — a resume and cover letter built around what this employer wants.

Get past ATS filters

Job summary

YO AI Labs is seeking Medical Evaluation Specialists, including medical students, residents, physicians, and qualified biomedical professionals, to contribute clinical expertise to evaluating and improving next-generation AI systems. You will create and validate challenging medical questions and answers designed to test advanced clinical reasoning.

Your work will help assess whether AI systems can accurately interpret medical evidence, synthesize complex information, and apply nuanced clinical

Qualifications

Medical training or clinical credential required.
Experience in research and source triangulation.
Strong attention to detail and written precision.
Excellent analytical thinking and clinical reasoning.
Experience with medical literature review and evidence synthesis.
Ability to develop high-quality medical questions.

Responsibilities

Create original, high-difficulty medical question-and-answer pairs covering diagnosis, pathophysiology, pharmacology, clinical guidelines, and clinical decision-making.
Develop questions that require synthesis, interpretation, and genuine medical reasoning.
Research and verify answers using primary literature, clinical guidelines, systematic reviews, and authoritative medical references.
Document the rationale behind answers and provide supporting citations.
Evaluate AI-generated responses for clinical accuracy, completeness, reasoning quality, and evidence adherence.
Identify questions that are too straightforward and refine them to increase complexity while staying clinically valid.
Review questions and answers for ambiguity, unsupported assumptions, factual errors, and inconsistencies.
Ensure all deliverables are written with clarity, precision, and defensibility.
Incorporate reviewer feedback and adapt work to evolving project guidelines and quality standards.
Contribute insights to establish rigorous benchmarks for medical AI evaluation.