Remote Medical Evaluation Designer for AI Benchmarks

YO AI Labs; Texas; Remote; USD 70.000 - 120.000; Teilzeit

Zusammenfassung

YO AI Labs seeks Medical Evaluation Specialists, including medical students, residents, physicians, and biomedical professionals, to contribute clinical expertise to evaluating and improving AI systems. You will create and validate high-difficulty medical questions and answers to test advanced clinical reasoning and evidence synthesis.

No AI experience required; your medical training, research skills, and clear writing are valued.

Qualifikationen

• Medical training or equivalent biomedical expertise.
• Strong written English and exceptional attention to detail.
• Ability to locate, interpret, and synthesize information from primary research and clinical guidelines.
• Experience writing medical questions, peer review, or related medical content evaluation is a plus.

Aufgaben

• Create original, high-difficulty medical question-and-answer pairs across diagnosis, pathophysiology, pharmacology, guidelines, and clinical decision-making.
• Develop questions that require synthesis, interpretation, and genuine medical reasoning, not just recall.
• Research and verify answers using primary literature, guidelines, reviews, and authoritative references.
• Document rationale and provide supporting citations.
• Evaluate AI-generated responses for accuracy, completeness, and quality of reasoning.
• Refine questions to increase complexity while maintaining validity.
• Review for ambiguity, errors, and inconsistencies; ensure clarity, precision, and defensibility.
• Incorporate feedback and adapt to evolving project guidelines and quality standards.
• Contribute insights to establish rigorous benchmarks for medical AI evaluation.

Kenntnisse

Medical Training

Research & Source Triangulation

Attention to Detail

Written Precision

Analytical Thinking

Clinical Reasoning

Medical Literature Review

Evidence Synthesis

Self-Direction & Reliability

Medical Question Development

Ausbildung

Medical training or biomedical expertise

Jobbeschreibung

Job Title: Medical Evaluation Specialist

Role Type: Contractor
Location: Remote

Job Overview

We are seeking Medical Evaluation Specialists, including medical students, residents, physicians, and qualified biomedical professionals, to contribute clinical expertise to a project focused on evaluating and improving next-generation AI systems.

In this role, you will create and validate challenging medical questions and answers designed to test advanced clinical reasoning. Your work will help assess whether AI systems can accurately interpret medical evidence, synthesize complex information, and apply nuanced clinical judgment.

No prior AI experience is required—your medical knowledge, research skills, and clinical reasoning are what matter most.

Scope of Work

• Create original, high-difficulty medical question-and-answer pairs covering areas such as diagnosis, pathophysiology, pharmacology, clinical guidelines, and clinical decision-making.
• Develop questions that require synthesis, interpretation, and genuine medical reasoning rather than simple factual recall.
• Research and verify answers using primary literature, clinical guidelines, systematic reviews, and authoritative medical references.
• Document the rationale behind answers and provide appropriate supporting citations.
• Evaluate AI-generated responses for clinical accuracy, completeness, reasoning quality, and adherence to available evidence.
• Identify questions that are too straightforward and refine them to increase complexity while maintaining clinical validity.
• Review questions and answers for ambiguity, unsupported assumptions, factual errors, and inconsistencies.
• Ensure all deliverables are written with clarity, precision, and defensibility.
• Incorporate reviewer feedback and adapt work to evolving project guidelines and quality standards.
• Contribute insights that help establish rigorous benchmarks for medical AI evaluation.

Required Skills

• Medical Training
• Research & Source Triangulation
• Attention to Detail
• Written Precision
• Analytical Thinking
• Clinical Reasoning
• Medical Literature Review
• Evidence Synthesis
• Self-Direction & Reliability
• Medical Question Development

Preferred Qualifications

• Current or recent medical training or clinical practice as a medical student, resident, or physician, or equivalent expertise in a relevant biomedical discipline.
• Demonstrated ability to locate, interpret, compare, and synthesize information from primary research and clinical guidelines.
• Strong written English skills with exceptional attention to detail.
• Ability to create engaging, well-structured, challenging, and clinically nuanced questions.
• Strong understanding of evidence-based medicine and clinical reasoning.
• Ability to work independently, manage deadlines, and consistently deliver high-quality work in a remote environment.
• Previous experience with medical question writing, peer review, clinical education, medical content evaluation, or research is a plus.
• Familiarity with AI/ML systems, AI evaluation, or medical AI applications is advantageous but not required.

Compensation Structure

Compensation is output-based, with contributors paid per task that meets the applicable project specifications and quality standards. The time required to complete each task may vary depending on experience and individual workflow.

Minimum submission requirements apply, and selected contributors are expected to complete a minimum number of tasks per week.

Start Timeline & Availability

Roles are typically filled within 48 hours. Selected contributors should be prepared to begin their first assignments within 24–48 hours of completing onboarding.