Generalist - Labeling

Mercor; Remote; CAD 80,000 - 100,000; Part time

Job summary

A talent marketplace for AI projects seeks detail-oriented generalists for data labeling and content evaluation in a remote setting. Responsibilities include evaluating AI responses and assigning preference rankings. Ideal candidates have prior experience in similar tasks, strong critical thinking skills, and attention to detail. Flexible part-time commitment of 10–20 hours/week is expected. Competitive pay ranges from $25–35/hour depending on experience.

Qualifications

• Prior experience in data labeling, content moderation, or preference ranking tasks.
• Excellent critical thinking and reading comprehension skills.
• Strong attention to detail and consistency across repetitive tasks.
• Comfort working with evolving guidelines and ambiguity.

Responsibilities

• Evaluate and compare AI-generated responses based on quality, coherence, and helpfulness.
• Assign preference rankings to pairs or sets of model outputs.
• Provide brief written explanations for ranking decisions when required.
• Flag edge cases or inconsistencies in task design or model output.

Skills

Critical thinking

Reading comprehension

Attention to detail

Ability to handle ambiguity

Data labeling or content evaluation experience

Job description

1. Role Overview

Mercor is collaborating with a leading AI lab on a short-term project focused on improving preference ranking models for conversational AI systems. We’re seeking detail-oriented generalists—ideally with prior experience in data labeling or content evaluation—to assess and rank model outputs across a variety of domains. This opportunity is well-suited for professionals comfortable with nuanced judgment tasks and working independently in a remote setup.

2. Key Responsibilities

• Evaluate and compare AI-generated responses based on quality, coherence, and helpfulness

• Assign preference rankings to pairs or sets of model outputs

• Follow detailed labeling guidelines and adjust based on evolving criteria

• Provide brief written explanations for ranking decisions when required

• Flag edge cases or inconsistencies in task design or model output

3. Ideal Qualifications

• Prior experience in data labeling, content moderation, or preference ranking tasks

• Excellent critical thinking and reading comprehension skills

• Comfort working with evolving guidelines and ambiguity

• Strong attention to detail and consistency across repetitive tasks

• Availability for regular part-time work on a weekly basis

4. More About the Opportunity

• Remote and asynchronous — set your own hours

• Expected commitment: 10–20 hours/week

• Flexible workload depending on your availability and performance

5. Compensation & Contract Terms

• $25–35/hour depending on experience and location

• Payments issued weekly via Stripe Connect

• This is a freelance engagement; you’ll be classified as an independent contractor

6. Application Process

• Submit your resume to get started

• Complete a short form to highlight your relevant experience

• You may be asked to complete a brief assessment to evaluate task fit

• Expect a response within 3–5 business days

7. About Mercor

• Mercor is a talent marketplace that connects top experts with leading AI labs and research organizations

• Our investors include Benchmark, General Catalyst, Adam D’Angelo, Larry Summers, and Jack Dorsey

• Thousands of professionals across law, engineering, research, and creative fields collaborate on contract-based projects shaping the future of AI