Senior STEM & Tech SME - AI Question Design & Rubrics

Joinkeywork; United States; Remote; USD 55,000 - 124,000; Full time

Job summary

Joinkeywork seeks a remote, hourly contractor to act as a STEM & Technical SME. You will craft expert-level questions, evaluate AI content, and design rigorous rubrics that specify required reasoning, steps, and partial credit.

You’ll critique rubrics and develop gold-standard content to improve AI models. Responsibilities emphasize creating challenging problems, assessing correctness, and maintaining high technical standards across disciplines.

Qualifications

• Bachelor’s degree in Mathematics, Physics, Engineering, CS, or a technical discipline.
• Strong professional English (C1+), able to write precise explanations and evaluation criteria.
• 3+ years of professional, academic, or industry experience in a technical domain.
• Demonstrated advanced expertise in a clearly defined technical field or specialization.
• Ability to create challenging technical questions that require multi-step reasoning.
• Strong ability to design detailed evaluation rubrics defining required reasoning, correct methodology, and partial-credit criteria.
• Ability to distinguish between correct answers and flawed reasoning.
• Attention to formulas, units, and edge cases.
• Experience with exam writing, grading, peer review, or assessment design is preferred.
• Prior experience with AI evaluation, RLHF, SFT, benchmarking is preferred.

Responsibilities

• Create expert-level technical questions that test genuine technical reasoning and expose weaknesses in advanced AI systems.
• Design evaluation rubrics with structured criteria defining what a correct response must contain.
• Define partial-credit standards, acceptable alternatives, and critical errors.
• Evaluate AI-generated solutions for correctness, reasoning quality, and technical precision.
• Identify hidden reasoning errors and invalid assumptions.
• Critique peer rubrics for clarity, completeness, and scoring consistency.
• Develop reference content: expert answers, explanations, critiques, and gold-standard content.
• Support AI training with questions, rubrics, evaluations, and reference-answer pairs.

Skills

English proficiency

Technical problem design

Rubric design

Question development

Evaluation criteria

Multi-step reasoning

Peer review

Critical thinking

Education

Bachelor’s degree or higher in a technical field

Job description

Joinkeywork seeks a remote, hourly contractor to act as a STEM & Technical SME. You will craft expert-level questions, evaluate AI content, and design rigorous rubrics that specify required reasoning, steps, and partial credit.

You’ll critique rubrics and develop gold-standard content to improve AI models. Responsibilities emphasize creating challenging problems, assessing correctness, and maintaining high technical standards across disciplines.