AI Model Evaluation Specialist — Remote, Contract

mercor; Australia; Remote; AUD 138,000 - 197,000; Part time

Job summary

Mercor is seeking a Business Domain Expert for an AI Model Evaluation Pilot on a contract basis. This remote role focuses on admin, marketing, HR, and accounting tasks within AI evaluation projects.

You will solve domain-specific tasks, review model outputs, and provide evidence-based feedback while meeting tight deadlines. The ideal candidate has 5+ years in admin/marketing/HR/accounting, is a working professional who can judge work effectively, and can produce clear rationales under short

Qualifications

• 5+ years of hands-on professional experience in Admin, Marketing, HR, or Accounting.
• Currently or recently practicing to judge work as a working professional would.
• Ability to write clear, specific, evidence-grounded rationales for judgments.
• Reliable within a short turnaround window.

Responsibilities

• Solve one realistic, domain-specific task using the provided source files.
• Review five model-generated outputs for the task. Rank them 1-5 with no ties, score each 0-100, and write evidence-based rationales citing specific parts of the output.
• Provide structured feedback on the task and study design.
• Work independently to meet deadlines within a short turnaround window.
• Use professional judgment as no rubric or golden answer is provided.

Job description

Mercor is seeking a Business Domain Expert for an AI Model Evaluation Pilot on a contract basis. This remote role focuses on admin, marketing, HR, and accounting tasks within AI evaluation projects.

You will solve domain-specific tasks, review model outputs, and provide evidence-based feedback while meeting tight deadlines. The ideal candidate has 5+ years in admin/marketing/HR/accounting, is a working professional who can judge work effectively, and can produce clear rationales under short