Internal Medicine AI Evaluation Specialist

Weekday 1; Canada; Remote; CAD 179,000 - 248,000; Part time

Job summary

Weekday 1 is seeking experienced Internal Medicine physicians to contribute to training and evaluating AI models focused on healthcare. This fully remote role involves developing realistic clinical scenarios and references, evaluating AI-generated responses, and providing feedback to improve AI reasoning in diagnosis and treatment planning.

The engagement is independent contractor-based, with a flexible schedule around 20 hours per week and weekly payments, targeting experienced clinicians

Qualifications

• Board-certified in Internal Medicine.
• Current, active, and unrestricted medical license.
• Final-year Internal Medicine resident; board-eligible for recent graduates.
• Fellows: board-certified or board-eligible in Internal Medicine or subspecialty.

Responsibilities

• Design realistic Internal Medicine clinical scenarios.
• Create attending-level reference solutions reflecting evidence-based guidelines.
• Evaluate AI-generated clinical responses using structured evaluation frameworks.
• Provide detailed written feedback to improve AI model performance and clinical reliability.
• Collaborate with clinical teams for onboarding and calibration across specialties.

Skills

Board-certified

Active medical license

Board-eligible

Education

Board-certified in Internal Medicine

Active medical license

Final-year Internal Medicine resident

Job description

This role is for one of our clients

Compensation: $130-$180 per hour

Join a leading AI research initiative focused on advancing healthcare-focused artificial intelligence. We are seeking experienced Internal Medicine physicians to contribute their clinical expertise toward training and evaluating next-generation AI models capable of sophisticated medical reasoning.

In this role, you will develop realistic clinical scenarios, evaluate AI-generated responses against evidence-based standards, and help improve how AI systems reason about diagnosis, treatment planning, and patient management within Internal Medicine.

We welcome Internal Medicine attending physicians, hospitalists, fellows, final-year residents, and other qualified Internal Medicine professionals who are passionate about applying their expertise to the future of healthcare AI.

Requirements

Key Responsibilities

Design Clinical Scenarios

• Create realistic Internal Medicine case studies based on everyday clinical practice.


• Develop prompts covering:
  
  
  • Diagnostic reasoning
  
  
  • Differential diagnosis
  
  
  • Treatment planning
  
  
  • Risk stratification
  
  
  • Evidence-based guideline adherence
  
  
  • Long-term patient management
  
  

Create Reference-Quality Responses

• Produce attending physician-level reference solutions that reflect current clinical best practices.


• Ensure recommendations align with established evidence-based guidelines.

Evaluate AI Performance

• Review and assess AI-generated clinical responses using structured evaluation frameworks.


• Identify reasoning gaps, inaccuracies, omissions, and opportunities for improvement.


• Provide detailed written feedback to improve AI model performance and clinical reliability.

Collaborate with Clinical Teams

• Participate in onboarding sessions and specialty calibration meetings.


• Work with multidisciplinary teams to ensure consistency, accuracy, and quality across medical evaluations.

Required Qualifications

Applicants should meet one of the following criteria:

Attending Physicians

• Board-certified in Internal Medicine.


• Current, active, and unrestricted medical license.

Resident Physicians

• Final-year Internal Medicine resident.


• Recent residency graduates must be board-eligible.

Fellows

• Board-certified or board-eligible in Internal Medicine or a related subspecialty.


• Current, active, and unrestricted medical license.

Professionals with expertise in Internal Medicine subspecialties are encouraged to apply, including:

• Cardiology


• Hematology & Oncology


• Endocrinology


• Nephrology


• Gastroenterology


• Pulmonology


• Other Internal Medicine subspecialties

Preferred Qualifications

• Strong knowledge of evidence-based medicine and clinical practice guidelines.


• Experience in clinical teaching, medical research, or quality improvement initiatives.


• Excellent written communication and analytical skills.


• Interest in artificial intelligence, clinical decision support, or healthcare technology.

Engagement Details

• Fully remote with 100% asynchronous work.


• Standard commitment of approximately 20 hours per week, with opportunities for additional hours based on project demand.


• Flexible working schedule.


• Weekly payments.


• Rolling onboarding with ongoing engagement opportunities.


• Compensation is determined by clinical experience, expertise, and seniority.

Why Join

• Help shape the future of AI-powered clinical decision-making.


• Apply your Internal Medicine expertise to improve advanced healthcare AI systems.


• Collaborate with leading AI researchers and experienced clinicians.


• Contribute to high-impact projects that advance safe, accurate, and evidence-based medical AI.

Equal Opportunity Statement

All qualified applicants will be considered without regard to legally protected characteristics. Reasonable accommodations are available upon request.

Contract Information

• Independent contractor engagement.


• Fully remote work completed on your own schedule.


• Projects may be extended, modified, or concluded based on business needs and individual performance.


• Work does not involve access to confidential or proprietary information from any employer, client, or institution.


• Weekly payments are processed through supported payment platforms based on approved work completed.


• Please note that visa sponsorship is not available for this opportunity.