Remote | Medicine Physician (MD/DO/Doctoral study/PhD)

24-Mag Llc; New York (NY); Remote; USD 138,000 - 207,000; Part time

Benefits offered by this job

Remote work

Job summary

24-MAG LLC invites licensed physicians currently in active clinical practice to contribute to advanced medical AI evaluation through clinical reasoning, scenario development, and expert review of model performance. This part-time remote engagement focuses on realistic medical problem evaluation and evidence-based practice alignment.

You will collaborate with research teams to design scenarios, assess clinical reasoning, and provide structured feedback to improve medical model benchmarks, with

Qualifications

• Licensed physician currently in active clinical practice.
• Strong understanding of evidence-based medicine.
• Ability to assess complex medical reasoning objectively.
• Excellent written and verbal communication skills.

Responsibilities

• Evaluate AI performance on realistic medical problems.
• Assess quality of clinical reasoning and decision-making.
• Identify gaps in medical knowledge or clinical judgement.
• Review model conclusions against evidence-based practice.
• Collaborate with research teams on medical AI evaluation.

Skills

Licensed physician

Clinical decision-making

Evidence-based medicine

Scientific writing

Remote collaboration

Education

MD/DO or equivalent

Job description

We are sharing a specialised part-time opportunity for licensed physicians in active clinical practice to contribute to advanced medical AI evaluation through clinical reasoning, scenario development, structured assessment, and expert review of model performance.

Selected physicians will work with research teams to evaluate how AI systems approach realistic medical problems, identify gaps in clinical knowledge and reasoning, and help develop evaluation methods that reflect the complexity and nuance of evidence-based clinical practice.

Key Responsibilities

Clinical Reasoning Evaluation

• Evaluate AI performance on realistic medical problems
• Assess the quality of clinical reasoning and decision-making
• Identify gaps in medical knowledge or clinical judgement
• Review model conclusions against evidence-based practice
• Distinguish significant clinical errors from minor limitations

Clinical Scenario Development

• Create realistic scenarios for medical AI evaluation
• Design cases requiring nuanced clinical judgement
• Incorporate diagnostic and management decision points
• Develop scenarios reflecting real-world clinical complexity
• Ensure cases meaningfully test model reasoning capabilities

Evaluation Framework Design

• Design systematic frameworks for medical AI assessment
• Define criteria for evaluating clinical reasoning quality
• Develop methods that capture nuance in clinical practice
• Support consistent assessment across repeated evaluation tasks
• Refine evaluation methods based on observed model performance

Medical Knowledge & Decision Analysis

• Assess model understanding of clinical concepts
• Evaluate diagnostic and treatment-related reasoning
• Identify unsupported assumptions or missing considerations
• Apply specialty-specific expertise where relevant
• Provide structured rationale for evaluation decisions

Research Collaboration & Feedback

• Collaborate with research teams on medical AI evaluation
• Communicate clinical findings clearly and concisely
• Provide actionable feedback on model strengths and limitations
• Contribute to strategies for improving medical model performance
• Support development of clinically meaningful benchmarks

Ideal Profile

• Licensed physician currently engaged in active clinical practice
• MD, DO, or equivalent medical qualification
• Physicians from any medical specialty may be considered
• Strong experience with clinical decision-making
• Solid understanding of evidence-based medicine
• Ability to assess complex medical reasoning objectively
• Strong analytical and critical-thinking capabilities
• Excellent written and verbal communication skills
• Comfortable explaining nuanced clinical decisions clearly
• Ability to identify gaps in medical knowledge or reasoning
• Interest in how AI can support clinical practice
• Comfortable contributing to structured model-evaluation workflows
• Strong problem-solving abilities
• Able to collaborate effectively with research teams
• Comfortable working independently in a remote environment
• Able to integrate project work around existing clinical commitments

Engagement Details

• Part-time remote engagement
• Flexible scheduling
• Commitment of up to 30 hours per week
• Initial project duration of approximately 1 month
• Extension may be available depending on performance and project fit
• Work will focus on clinical reasoning evaluation, scenario development, and medical benchmark design
• Physicians from any clinical specialty may be considered
• Scheduling is intended to accommodate ongoing clinical commitments
• Compensation is not specified in the source materials
• Work must be completed without using confidential, proprietary, patient-identifiable, protected health, unpublished clinical, or otherwise restricted information belonging to any patient, employer, healthcare organisation, research institution, client, or other third party

About the Platform

This opportunity is available through 24-MAG LLC. We connect experienced professionals with remote consulting opportunities across technical, evaluation, and project-based workstreams.