AI Product Design Evaluator & Criteria Architect

Mercor; New York (NY); Remote; USD 90,000 - 150,000; Full time

Job summary

Mercor is partnering with a leading AI research organization to engage experienced UI/UX and product designers for a project that evaluates how well AI systems perform real-world digital product design work. You will define what excellent work looks like by designing task-specific grading criteria and scoring samples with rigorous, written justifications, guiding evaluation rather than delivering designs.

You will contribute to an objective assessment framework and collaborate with senior

Qualifications

• 5+ years of professional experience in product design or UI/UX for real digital products.
• Experience at top digital-product studios or major product-led companies with portals/dashboards.
• Experience across tracks: Product/UI/Design Systems Designer or UX Lead or similar.
• Deep fluency in interaction/visual design, design systems, responsive layouts, end-to-end product flows, prototyping and handoff in Figma.
• Excellent written communication and ability to justify critique.
• Detail-oriented and open to peer review.
• Experience with AI training/evaluation a plus.

Responsibilities

• Design task-specific grading criteria for real-world digital product and UX deliverables (design systems and component libraries, responsive UI layouts and dashboards, end-to-end product flows such as onboarding, checkout, and booking, logged-in customer portals, UX writing and product content, and design-to-engineering/production handoff)
• Score AI-generated and human work samples against those criteria, with detailed written justifications for every score
• Apply consistent, evidence-based judgment so that scores are reproducible and defensible
• Incorporate structured feedback from senior reviewers and iterate quickly on your work

Skills

Interaction design

Visual design

Design systems

End-to-end product flows

Written communication

Tools

Figma

Job description

Mercor is partnering with a leading AI research organization to engage experienced UI/UX and product designers for a project that evaluates how well AI systems perform real-world digital product design work. You will define what excellent work looks like by designing task-specific grading criteria and scoring samples with rigorous, written justifications, guiding evaluation rather than delivering designs.

You will contribute to an objective assessment framework and collaborate with senior