Remote

24-MAG; New York (NY); Remote; USD 96,000 - 165,000; Part time

Benefits offered by this job

Remote work

Job summary

24-MAG LLC seeks licensed physicians in active clinical practice for a part-time remote engagement to evaluate medical AI through clinical reasoning, scenario development, and model performance reviews.

Selected physicians will collaborate with research teams to develop evaluation methods and benchmarks reflecting evidence-based practice, with flexible scheduling and project-based terms.

Qualifications

• Licensed physician currently in active clinical practice.
• MD, DO, or equivalent medical qualification.
• Physicians from any medical specialty may be considered.
• Ability to assess clinical decision-making objectively.

Responsibilities

• Clinical Reasoning Evaluation: evaluate AI performance on medical problems and assess reasoning quality.
• Clinical Scenario Development: design realistic cases testing nuanced clinical judgment.
• Evaluation Framework Design: create structured evaluation criteria for clinical practice.
• Medical Knowledge & Decision Analysis: review concepts, diagnoses and treatments with evidence-based reasoning.
• Research Collaboration & Feedback: provide actionable input to improve model performance.

Skills

Clinical decision-making

Evidence-based medicine

Critical thinking

Written communication

Verbal communication

Remote collaboration

Education

MD/DO or equivalent

Job description

We are sharing a specialised part-time opportunity for licensed physicians in active clinical practice to contribute to advanced medical AI evaluation through clinical reasoning, scenario development, structured assessment, and expert review of model performance.

Selected physicians will work with research teams to evaluate how AI systems approach realistic medical problems, identify gaps in clinical knowledge and reasoning, and help develop evaluation methods that reflect the complexity and nuance of evidence-based clinical practice.

Key Responsibilities

Clinical Reasoning Evaluation

• Evaluate AI performance on realistic medical problems
• Assess the quality of clinical reasoning and decision-making
• Identify gaps in medical knowledge or clinical judgement
• Review model conclusions against evidence-based practice
• Distinguish significant clinical errors from minor limitations

Clinical Scenario Development

• Create realistic scenarios for medical AI evaluation
• Design cases requiring nuanced clinical judgement
• Incorporate diagnostic and management decision points
• Develop scenarios reflecting real-world clinical complexity
• Ensure cases meaningfully test model reasoning capabilities

Evaluation Framework Design

• Design systematic frameworks for medical AI assessment
• Define criteria for evaluating clinical reasoning quality
• Develop methods that capture nuance in clinical practice
• Support consistent assessment across repeated evaluation tasks
• Refine evaluation methods based on observed model performance

Medical Knowledge & Decision Analysis

• Assess model understanding of clinical concepts
• Evaluate diagnostic and treatment-related reasoning
• Identify unsupported assumptions or missing considerations
• Apply specialty-specific expertise where relevant
• Provide structured rationale for evaluation decisions

Research Collaboration & Feedback

• Collaborate with research teams on medical AI evaluation
• Communicate clinical findings clearly and concisely
• Provide actionable feedback on model strengths and limitations
• Contribute to strategies for improving medical model performance
• Support development of clinically meaningful benchmarks

Ideal Profile

• Licensed physician currently engaged in active clinical practice
• MD, DO, or equivalent medical qualification
• Physicians from any medical specialty may be considered
• Strong experience with clinical decision-making
• Solid understanding of evidence-based medicine
• Ability to assess complex medical reasoning objectively
• Strong analytical and critical-thinking capabilities
• Excellent written and verbal communication skills
• Comfortable explaining nuanced clinical decisions clearly
• Ability to identify gaps in medical knowledge or reasoning
• Interest in how AI can support clinical practice
• Comfortable contributing to structured model-evaluation workflows
• Strong problem-solving abilities
• Able to collaborate effectively with research teams
• Comfortable working independently in a remote environment
• Able to integrate project work around existing clinical commitments

Engagement Details

• Part-time remote engagement
• Flexible scheduling
• Commitment of up to 30 hours per week
• Initial project duration of approximately 1 month
• Extension may be available depending on performance and project fit
• Work will focus on clinical reasoning evaluation, scenario development, and medical benchmark design
• Physicians from any clinical specialty may be considered
• Scheduling is intended to accommodate ongoing clinical commitments
• Compensation is not specified in the source materials
• Work must be completed without using confidential, proprietary, patient-identifiable, protected health, unpublished clinical, or otherwise restricted information belonging to any patient, employer, healthcare organisation, research institution, client, or other third party

About the Platform

This opportunity is available through 24-MAG LLC. We connect experienced professionals with remote consulting opportunities across technical, evaluation, and project-based workstreams.