Medical AI Clinical Reasoning Evaluator

OpenTrain AI, Inc.; United States; Remote; USD 83,000 - 152,000; Part time

Job summary

OpenTrain AI, Inc. is seeking licensed physicians to participate in remote, flexible AI training evaluations. You will assess how AI systems perform on medical problems and clinical decision-making tasks, applying evidence-based medicine and your practical medical judgment.

You will work with AI researchers to clarify model strengths and gaps, design evaluation criteria, and develop clinical scenarios. Hours start at 20 per week (up to 30) for about one month, with potential extensions based on

Qualifications

• Active medical license in good standing.
• Current clinical practice experience.
• Strong foundation in evidence-based medicine.
• Excellent analytical and communication skills.

Responsibilities

• Design systematic evaluation frameworks for medical AI systems.
• Create clinical scenarios that test AI reasoning and decision-making.
• Build assessment methods grounded in clinical practice.
• Identify gaps in AI medical knowledge and clinical reasoning.
• Apply clinical judgment and evidence-based medicine to evaluate AI behavior.
• Collaborate with AI researchers to improve model performance.

Skills

Analytical thinking

Communication skills

Clinical decision-making

Evidence-based medicine

Education

Active medical license

Job description

The work

You will assess how AI systems perform on medical problems and clinical decision-making tasks. The work uses your practical medical judgment and evidence-based medicine to identify gaps in AI knowledge and reasoning.

You will also work with AI researchers to clarify where models perform well or fall short. Experience turning complex clinical judgment into clear evaluation criteria, scenarios, or structured assessments is useful.

• Design systematic evaluation frameworks for medical AI systems.
• Create clinical scenarios that test AI reasoning and decision-making.
• Build assessment methods grounded in the details of clinical practice.
• Identify gaps in AI medical knowledge and clinical reasoning.
• Apply clinical judgment and evidence-based medicine to evaluate AI behavior.
• Collaborate with AI researchers to improve model performance.

What it pays and takes

This is a remote, flexible engagement for a practicing physician. Scheduling can be arranged around ongoing clinical commitments, and the engagement may be extended based on performance and fit.

• Pay: Not specified in the listing.
• Hours: 20 or more hours per week, up to 30 hours per week.
• Duration: One month, with potential extensions based on performance and fit.
• Location: Remote and open worldwide.
• Work type: Contractor and part-time.
• Language: English.
• Required: An active license to practice medicine in any specialty.
• Required: Current clinical practice experience and experience with clinical decision-making.
• Required: A strong foundation in evidence-based medicine.
• Required: Strong analytical and communication skills.
• Required: Interest in using AI in clinical practice.
• Helpful: Familiarity with medical AI, clinical decision support, or evaluating reasoning quality.

About AI training work

AI training is the human work behind systems that learn from examples, including reviewing model answers and evaluating reasoning. OpenTrain AI handles this engagement so experienced medical professionals can apply their knowledge to help improve AI systems.