Generalist | $40/hr Remote | Mercor

Crossing Hurdles; United States; Remote; USD 48,216 - 55,104; Part time

Job summary

A leading evaluation organization seeks a Generalist Evaluator Expert for a remote, flexible contract role. This position involves designing prompts for language model evaluations, defining standards, and conducting assessments. Ideal candidates should have strong writing and critical thinking skills, familiarity with AI tools like ChatGPT, and a BS or BA degree. This mid-senior level role offers flexible hours (10-40/week) and competitive compensation of $35-$40/hour.

Qualifications

• Strong writing skills and critical thinking necessary for prompt design.
• Ability to work independently while meeting deadlines.
• Significant familiarity with language tools like ChatGPT.

Responsibilities

• Design and optimize detailed prompts for AI model evaluations.
• Define evaluation standards and develop comprehensive rubrics.
• Conduct testing and grading of AI model outputs.

Skills

Strong writing skills

Critical thinking

Instructional clarity

Familiarity with ChatGPT

Ability to work independently

Education

BS or BA degree from a reputable institution

Job description

Generalist Evaluator Expert

Organization: Mercor

Referral Partner: Crossing Hurdles

Type: Hourly Contract

Compensation: $35–$40/hour

Location: Remote

Duration: Approximately 1 month

Commitment: 10-40 hours/week, flexible and asynchronous

Role Responsibilities

• Design and optimize detailed prompts with multiple constraints and clear instructions for language model evaluation
• Define and document evaluation standards and develop comprehensive rubrics for assessing correct responses in consumer contexts
• Conduct testing and grading by running prompts through AI models and assessing outputs against expectations
• Support benchmarking and quality assurance to ensure prompt rigor and consistency before integration into official evaluation sets
• Collaborate with team members in quality assurance review processes to maintain evaluation reliability

Requirements

• BS or BA degree from a reputable institution (completed or in progress)
• Strong writing skills, critical thinking, and instructional clarity
• Ability to work independently and meet deadlines
• Significant familiarity with ChatGPT or similar language tools for personal or professional use
• Experience in teaching or research is preferred
• Access to a desktop or laptop computer (Chromebooks are not supported)

Application Process

• Upload resume
• AI interview based on your resume (15 min)

Seniority Level: Mid-Senior level

Employment Type: Contract

Job Function: Quality Assurance, Writing/Editing, and Research

Industries: E-Learning Providers, Education, and Information Services