Healthcare Specialist – Rubric Design

Joinkeywork

United States

Remote

USD 96,000 - 165,000

Part time

14 days+

A complete application in a minute

Tailored resume and cover letter, ready to send.

Get past ATS filters

Job summary

SME Careers is seeking an experienced Healthcare SME for an hourly remote contractor role. You will create expert healthcare questions, design scoring rubrics, evaluate AI-generated content, and provide precise feedback to improve clinical accuracy and reliability.

The role emphasizes rubric design, expert evaluation, and high-level clinical reasoning. Applicants must have clinical credentials, strong English, and at least 3 years of experience in medicine or a related field, with a track record

Qualifications

  • Professional degree, license, or recognized qualification in medicine or a healthcare discipline.
  • Strong English proficiency at least C1.
  • 3+ years of professional or clinical experience; senior-level preferred.
  • Ability to write detailed evaluation rubrics defining required facts, reasoning steps, caveats, acceptable alternatives, and critical clinical errors.

Responsibilities

  • Create expert healthcare questions to expose limitations in advanced AI systems.
  • Design detailed rubrics defining components of accurate, clinically appropriate answers.
  • Identify critical criteria distinguishing essential clinical reasoning from supplementary information.
  • Evaluate AI outputs for factual accuracy, reasoning, safety, completeness, and professional quality.
  • Identify subtle failures such as missing considerations, unsupported assumptions, or clinically important omissions.
  • Critique peer rubrics for ambiguity, redundancy, or missing criteria.
  • Develop reference answers and gold-standard content for RL and SFT workflows.
  • Support AI training with questions, rubrics, evaluations, and feedback.

Skills

Clinical expertise
English proficiency
Question design
Rubric writing
AI evaluation experience

Education

Medical degree or equivalent

Job description

In this hourly, remote contractor role, you will work as a Healthcare Subject Matter Expert (SME) to create challenging, expert-level healthcare questions, evaluate AI-generated clinical content, and design high-quality rubrics defining what an accurate, complete, and professionally appropriate answer should contain.

You will use your clinical or professional expertise to identify areas where frontier AI models may provide incomplete, inaccurate, poorly reasoned, overconfident, or clinically inadequate responses. You will write specialized questions within your field, develop detailed scoring criteria, critique rubrics created by other healthcare professionals, and provide precise written feedback.

A major focus of this role is rubric design and expert evaluation. You will define essential clinical concepts, reasoning steps, relevant caveats, acceptable alternative approaches, significant omissions, and critical errors that distinguish superficial answers from expert-level responses.

This role is with SME Careers, a fast-growing AI Data Services company and subsidiary of SuperAnnotate, delivering training data for many of the world’s largest AI companies and foundation-model labs. Your healthcare expertise directly helps improve the world’s premier AI models by making their responses more accurate, clinically appropriate, rigorous, and reliable.

Responsibilities
  • Create expert healthcare questions: Develop specialized questions capable of exposing limitations in advanced AI systems.
  • Design detailed rubrics: Define the components required for an accurate, complete, clinically appropriate answer.
  • Identify critical criteria: Distinguish essential clinical reasoning from supplementary information.
  • Evaluate AI responses: Assess model outputs for factual accuracy, reasoning, safety, completeness, and professional quality.
  • Identify subtle failures: Detect missing considerations, unsupported assumptions, inappropriate certainty, and clinically important omissions.
  • Critique peer rubrics: Identify ambiguity, redundancy, incorrect expectations, missing criteria, or poor scoring design.
  • Develop reference answers: Produce expert examples, explanations, critiques, and gold-standard content.
  • Support AI training: Generate questions, rubrics, evaluations, critiques, and reference-answer pairs for RL and SFT workflows.
  • Maintain clinical quality: Ensure content reflects appropriate terminology, professional reasoning, and evidence-aware judgment.
Requirements
  • Professional degree, license, credential, or recognized qualification in medicine, dentistry, pharmacy, nursing, or another healthcare discipline.
  • Strong professional proficiency in English, minimum C1.
  • 3+ years of professional or clinical experience; senior and specialist-level professionals are strongly preferred.
  • Demonstrated expertise in a clearly defined specialty or subspecialty.
  • Ability to develop challenging clinical or professional questions that require genuine expertise and nuanced judgment.
  • Strong ability to write detailed evaluation rubrics defining required facts, reasoning steps, caveats, acceptable alternatives, and critical clinical errors.
  • Ability to distinguish between minor omissions and errors that materially affect clinical quality or safety.
  • Comfortable reviewing and critiquing rubrics created by other healthcare professionals.
  • Ability to evaluate AI outputs for accuracy, completeness, reasoning quality, safety, calibration, and clinical relevance.
  • Experience with medical education, examination writing, competency assessment, clinical guidelines, teaching, peer review, or quality assurance is strongly preferred.
  • Prior experience with AI evaluation, RLHF, SFT, data annotation, model benchmarking, or rubric-based scoring is preferred.
  • Reliable, self-directed, and capable of producing consistent written work.
Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Healthcare Specialist – Rubric Design
Healthcare Specialist – Rubric Design

SuperAnnotate AI, Inc. • United States

Remote
USD 83,000 - 138,000
Healthcare SME: Rubric Design & AI Evaluation (Remote)
Healthcare SME: Rubric Design & AI Evaluation (Remote)

SuperAnnotate AI, Inc. • United States

Remote
USD 83,000 - 138,000
Medical General Expert
Medical General Expert

SME.careers • United States

Remote
USD 55,000 - 110,000
Healthcare AI Rubric Evaluation Specialist
Healthcare AI Rubric Evaluation Specialist

OpenTrain AI, Inc. • United States

Hybrid
USD 114,000 - 162,000
Remote Healthcare Rubric Designer & AI Evaluator
Remote Healthcare Rubric Designer & AI Evaluator

Joinkeywork • United States

Remote
USD 96,000 - 165,000
Remote Healthcare AI Rubric & Evaluation Specialist
Remote Healthcare AI Rubric & Evaluation Specialist

OpenTrain AI, Inc. • United States

Hybrid
USD 114,000 - 162,000
STEM & Technical Specialist – Rubric Design
STEM & Technical Specialist – Rubric Design

Joinkeywork • United States

Remote
USD 55,000 - 124,000
Medical Doctor (Specialist)
Medical Doctor (Specialist)

SME.careers • United States

Remote
USD 165,000 - 303,000
Nurse
Nurse

SME.careers • United States

Remote
USD 83,000 - 152,000
STEM & Technical Specialist – Rubric Design
STEM & Technical Specialist – Rubric Design

SuperAnnotate AI, Inc. • United States

Remote
USD 114,000 - 162,000