AI Project Reviewer (CONTRACT)

Jobless; Northern (KY); Hybrid; USD 103,000 - 172,000; Full time

Benefits offered by this job

Flexible engagement

20 hours weekly

Job summary

MasterClass is seeking a freelance AI Project Reviewer to evaluate learner submissions against rubrics and provide actionable feedback. You will collaborate with product and learning design teams to calibrate assessment approaches, ensuring fair, accurate, and valuable feedback for members.

This temporary role offers about 20 hours per week and a flexible engagement. Ideal candidates have 5+ years in assessment or educational feedback, a proven ability to apply rubrics consistently, and strong

Qualifications

• 5+ years of experience in assessment, evaluation, or educational feedback roles.
• Ability to consistently apply detailed rubrics across multiple submissions.
• Strong written communication and ability to articulate reasoning clearly.
• Experience providing constructive feedback to support learner growth.
• Ability to work independently while maintaining high standards.
• Willingness to refrain from using AI during evaluations and rationales.

Responsibilities

• Review and evaluate learner projects against detailed rubrics with consistent pass/fail judgments.
• Write clear, actionable feedback to help learners improve.
• Collaborate with learning design and product teams to refine rubrics and criteria.
• Compare human and AI assessment outcomes to identify calibration needs.
• Document reasoning behind evaluation decisions to support consistency.

Skills

Attention to detail

Written communication

Rubric-based assessment

Independent work

Constructive feedback

Job description

Who we are:

MasterClass is the streaming platform where the world's best come together so anyone, anywhere, can access and be inspired by their knowledge and stories. We put you in the room with the creators, thinkers, makers and leaders who have changed the world, so that you can change yours.

Members get unprecedented access to 200+ instructors and classes across a wide variety of fields, including Arts & Entertainment, Business, Design & Style, Sports & Gaming, Writing and more. Step into Nas' recording studio and Gordon Ramsay's kitchen, and go behind the big screen with James Cameron. Design your career with Elaine Welteroth, get ready to win with Lewis Hamilton, perfect your pitch with Shonda Rhimes and discover your inner negotiator with Chris Voss.

Role Overview:

As a freelance AI Project Reviewer, you will be responsible for evaluating learner submissions against detailed rubrics for a new product. You will apply consistent judgment to determine pass/fail outcomes and provide clear, constructive feedback to help learners improve. Working alongside our product and learning design teams, you’ll help calibrate and validate assessment approaches to ensure fair, accurate, and valuable feedback for our members.

This is a temporary role with flexible engagement based on project needs. ~20 hours per week.

What You Will Do:

• Review and evaluate projects against detailed rubrics, providing consistent pass/fail judgments.
• Write clear, actionable feedback that helps learners understand areas for improvement.
• Collaborate with the learning design and product team to refine rubrics and assessment criteria.
• Compare human and AI assessment outcomes to identify areas for calibration.
• Document reasoning behind evaluation decisions to support assessment consistency

About You:

• 5+ years of experience in assessment, evaluation, or educational feedback roles.
• Proven ability to consistently apply detailed rubrics across multiple submissions.
• Strong written communication skills with the ability to articulate reasoning clearly.
• Experience providing constructive feedback to support learner growth.
• Ability to work independently while maintaining high standards.
• Willingness to refrain from using AI during evaluations and written rationales.
• Experience in roles such as Mentor or Coach in educational settings, Educational Evaluator or Teaching Assistant (TA), Consultant in AI or learning design, or Instructor with experience in grading and providing feedback.

You’re a Great Fit If You:

• Are passionate about education and committed to helping learners succeed.
• Have a keen eye for detail and take pride in maintaining high standards in evaluation.
• Enjoy collaborating with cross-functional teams to enhance assessment quality.
• Are comfortable making nuanced judgments while ensuring consistency.
• Have experience working independently and managing your time efficiently.
• Are curious and open to learning more about AI-driven evaluation methods.

Hourly Range: $75-$125 USD

The hourly range listed is for NYC/CA/WA. As a company, we have a location based strategy, which means the disclosed range estimate has not been adjusted for any applicable geographic differential associated with the location where the position may be filled.

At MasterClass, we believe we put our best work forward when our employees bring together ideas that are diverse in thought. We are proud to be an equal opportunity workplace and are committed to equal employment opportunity regardless of race, color, religion, national origin, age, sex, marital status, ancestry, physical or mental disability, genetic information, veteran status, gender identity or expression, sexual orientation, or any other characteristic protected by applicable federal, state or local law. In addition, MasterClass will provide reasonable accommodations for qualified individuals with disabilities. If you have a disability or special need, we would like to know how we can better accommodate you.