AI Product Design Evaluator

Mercor; San Francisco (CA); Remote; USD 120,000 - 180,000; Full time

Job summary

Mercor partners with a leading AI research organization to define precise grading criteria for real-world digital product design work. You will specify what constitutes excellent work and score samples with rigorous, well-reasoned written justifications.

You will apply structured, evidence-based judgment to both AI-generated and human outputs, ensuring scores are reproducible and defensible, and iterating quickly on feedback from senior reviewers.

Qualifications

• 5+ years of professional experience in product design, UI/UX, or design systems for real digital products.
• Experience at leading digital-product studios or major product-led companies.
• Fluency across tracks: Product/UI/Design Systems Designer, UX Lead, UX Writer, or Digital Analyst.
• Strong written communication; ability to critique work clearly.
• Detail-oriented with calibrated, reproducible judgments.
• Experience with AI training or human-data projects is a plus.

Responsibilities

• Design precise, task-specific grading criteria for real-world digital product and UX deliverables.
• Score AI-generated and human work samples with detailed written justifications.
• Apply consistent, evidence-based judgment for reproducible scores.
• Incorporate feedback from senior reviewers and iterate quickly.

Skills

Product design

UI/UX

Design systems

Prototyping

Written communication

AI evaluation

Tools

Figma

Job description

Mercor partners with a leading AI research organization to define precise grading criteria for real-world digital product design work. You will specify what constitutes excellent work and score samples with rigorous, well-reasoned written justifications.

You will apply structured, evidence-based judgment to both AI-generated and human outputs, ensuring scores are reproducible and defensible, and iterating quickly on feedback from senior reviewers.