AI Model Evaluation Specialist — Remote Contract

mercor; United States; Remote; USD 83,000 - 110,000; Full time

Job summary

Mercor is seeking a seasoned Business Domain Expert to pilot AI model evaluation across admin, marketing, HR and accounting workflows. This remote contractor role focuses on judging outputs with evidence-based rationales and delivering structured feedback within tight turnaround times.

You will review model-generated results, document reasoning, and provide design feedback to improve study quality. Ideal candidates have 5+ years in the listed domains and can work independently in a fast-paced AI

Qualifications

• 5+ years of hands-on experience in Admin, Marketing, Human Resources, or Accounting.
• Currently or recently practicing to judge work as a working professional would.
• Ability to write a clear, specific, evidence-grounded rationale for a judgment.
• Reliable within a short turnaround window.

Responsibilities

• Sign an NDA before receiving any materials.
• Solve one realistic, domain-specific task using the source files provided.
• Review five model-generated outputs for that task. Rank them 1-5 with no ties, score each 0-100, and write evidence-based rationales citing specific parts of the output.
• Provide structured feedback on the task itself and on the study design.
• Work independently to complete the task within a short turnaround window.

Skills

Admin

Marketing

Human Resources

Accounting

Job description

Mercor is seeking a seasoned Business Domain Expert to pilot AI model evaluation across admin, marketing, HR and accounting workflows. This remote contractor role focuses on judging outputs with evidence-based rationales and delivering structured feedback within tight turnaround times.

You will review model-generated results, document reasoning, and provide design feedback to improve study quality. Ideal candidates have 5+ years in the listed domains and can work independently in a fast-paced AI