Remote Healthcare Rubric Designer & AI Evaluator

Joinkeywork; United States; Remote; USD 96,000 - 165,000; Part time

Job summary

SME Careers is seeking an experienced Healthcare SME for an hourly remote contractor role. You will create expert healthcare questions, design scoring rubrics, evaluate AI-generated content, and provide precise feedback to improve clinical accuracy and reliability.

The role emphasizes rubric design, expert evaluation, and high-level clinical reasoning. Applicants must have clinical credentials, strong English, and at least 3 years of experience in medicine or a related field, with a track record

Qualifications

• Professional degree, license, or recognized qualification in medicine or a healthcare discipline.
• Strong English proficiency at least C1.
• 3+ years of professional or clinical experience; senior-level preferred.
• Ability to write detailed evaluation rubrics defining required facts, reasoning steps, caveats, acceptable alternatives, and critical clinical errors.

Responsibilities

• Create expert healthcare questions to expose limitations in advanced AI systems.
• Design detailed rubrics defining components of accurate, clinically appropriate answers.
• Identify critical criteria distinguishing essential clinical reasoning from supplementary information.
• Evaluate AI outputs for factual accuracy, reasoning, safety, completeness, and professional quality.
• Identify subtle failures such as missing considerations, unsupported assumptions, or clinically important omissions.
• Critique peer rubrics for ambiguity, redundancy, or missing criteria.
• Develop reference answers and gold-standard content for RL and SFT workflows.
• Support AI training with questions, rubrics, evaluations, and feedback.

Skills

Clinical expertise

English proficiency

Question design

Rubric writing

AI evaluation experience

Education

Medical degree or equivalent

Job description

SME Careers is seeking an experienced Healthcare SME for an hourly remote contractor role. You will create expert healthcare questions, design scoring rubrics, evaluate AI-generated content, and provide precise feedback to improve clinical accuracy and reliability.

The role emphasizes rubric design, expert evaluation, and high-level clinical reasoning. Applicants must have clinical credentials, strong English, and at least 3 years of experience in medicine or a related field, with a track record