Remote Healthcare AI Rubric & Evaluation Specialist

OpenTrain AI, Inc.; United States; Hybrid; USD 114,000 - 162,000; Part time

Job summary

OpenTrain AI, Inc. is seeking a remote, part-time contractor with healthcare expertise to help train AI by creating expert healthcare content and judging clinical responses. You will write challenging questions and scoring rubrics, and evaluate AI-generated clinical content for accuracy and safety.

The role requires strong clinical judgment and professional English writing skills, with focus on accuracy, completeness, safety, and relevance.

Qualifications

• Ability to write difficult questions and rubrics.
• Assess clinical reasoning and safety in responses.
• Familiarity with medical guidelines and education methods.

Responsibilities

• Write challenging healthcare questions that reveal limitations in AI responses.
• Create scoring rubrics covering essential concepts, reasoning steps, caveats, and alternatives.
• Evaluate AI-generated clinical content for accuracy, completeness, safety, and relevance.
• Review rubrics written by other healthcare professionals.
• Develop expert reference answers and explanations for reinforcement learning.

Skills

Healthcare content writing

Clinical evaluation

Rubric development

Question writing

Quality assessment

Education

Professional degree or healthcare credential

Job description

The work

You will help train AI by creating expert healthcare content and judging the quality of clinical responses. Your work will focus on accuracy, completeness, safety, reasoning, uncertainty, and professional relevance.

• Write challenging healthcare questions that reveal limitations in advanced AI responses.
• Create scoring rubrics covering essential concepts, reasoning steps, caveats, acceptable alternatives, and critical errors.
• Evaluate AI-generated clinical content for factual accuracy, completeness, safety, reasoning quality, calibration, and professional relevance.
• Distinguish minor omissions from problems that could materially affect clinical quality or safety.
• Review rubrics written by other healthcare professionals.
• Write expert reference answers, explanations, critiques, and gold-standard content for reinforcement learning and supervised fine-tuning work.

What it pays and takes

This is a remote, part-time contractor role for healthcare professionals in the United States. The work requires careful clinical judgment and strong professional writing in English.

• Pay: Up to $100 per hour.
• Time: 20 or more hours per week.
• Location: United States only.
• Language: Professional English proficiency at C1 level or higher.
• Education or credential: A professional degree, license, credential, or recognized qualification in medicine, dentistry, pharmacy, nursing, or another healthcare discipline.
• Experience: At least three years of professional or clinical experience in a clearly defined healthcare specialty or subspecialty.
• Skills: Ability to write difficult questions, build detailed rubrics, assess clinical reasoning and safety, identify important omissions, and recognize acceptable alternative approaches.
• Helpful background: Medical education, examination writing, competency assessment, clinical guidelines, teaching, peer review, quality assurance, AI evaluation, reinforcement learning from human feedback, supervised fine-tuning, data annotation, model benchmarking, or rubric-based scoring.

How it works

Apply on OpenTrain with your resume, then complete the application on the hiring site.

About AI training work

AI training is the human work behind systems that learn from examples, including writing questions, reviewing answers, and rating model responses. Healthcare specialists are needed because accurate, safe evaluation depends on professional knowledge and careful judgment.