Content Evaluator - Fully Remote

Mercor; United Kingdom; Remote; GBP 60,000 - 80,000; Part time

Job summary

A leading AI research talent agency is seeking a Freelance Preference Ranking Model Evaluator to assess AI-generated responses for quality and coherence. This remote position requires prior experience in data labeling or content moderation and strong critical thinking skills. Candidates should be detail-oriented and comfortable working with evolving guidelines. Flexible hours are available at $25-$35 per hour.

Qualifications

• Prior experience in data labeling, content moderation, or preference ranking tasks.
• Comfort working with evolving guidelines and ambiguity.
• Availability for regular part-time work on a weekly basis.

Responsibilities

• Evaluate and compare AI-generated responses based on quality, coherence, and helpfulness.
• Assign preference rankings to pairs or sets of model outputs.
• Provide brief written explanations for ranking decisions when required.

Skills

Critical thinking

Reading comprehension

Attention to detail

Job description

Base pay range

$25.00/hr - $35.00/hr

About The Job

Mercor connects elite creative and technical talent with leading AI research labs.
Headquartered in San Francisco, our investors include Benchmark, General Catalyst, Peter Thiel, Adam D'Angelo, Larry Summers, and Jack Dorsey.

Position: Preference Ranking Model Evaluator
Type: Freelance
Compensation: $25–$35/hour
Location: Remote
Commitment: 10–20 hours/week

Role Responsibilities

• Evaluate and compare AI-generated responses based on quality, coherence, and helpfulness.
• Assign preference rankings to pairs or sets of model outputs.
• Follow detailed labeling guidelines and adjust based on evolving criteria.
• Provide brief written explanations for ranking decisions when required.
• Flag edge cases or inconsistencies in task design or model output.
• Work independently and asynchronously to meet deadlines while improving AI model performance.

Qualifications

• Prior experience in data labeling, content moderation, or preference ranking tasks.
• Excellent critical thinking and reading comprehension skills.
• Comfort working with evolving guidelines and ambiguity.
• Strong attention to detail and consistency across repetitive tasks.
• Availability for regular part-time work on a weekly basis.

Compensation & Legal

• Payments issued weekly via Stripe Connect.
• Classified as an independent contractor.

Application Process (Takes 20–30 mins to complete)

• Submit your resume.
• Complete a short form to highlight your relevant experience.
• You may be asked to complete a brief assessment to evaluate task fit.

Resources & Support

• For details about the interview process and platform information, please check: https://talent.docs.mercor.com/welcome/welcome
• For any help or support, reach out to: [e-mail]

PS: Our team reviews applications daily. Please complete your AI interview and application steps to be considered for this opportunity.