Health Ai Conversation Evaluator

Mercor; San Francisco (CA); Hybrid; GBP 21,000 - 166,000; Part time

Job summary

Mercor is seeking a Generalist Annotator for Health AI Conversation Quality Evaluation. This remote contract role pays $20–$160/hour. You will independently review conversations (1–7 turns) and provide concise, rubric-based feedback to improve AI interactions.

Strong written English and the ability to explain judgments in 2–4 sentences are required. Ability to read carefully and adhere to a detailed rubric is essential. Remote work is offered in the UK and beyond.

Qualifications

• Strong written English required.
• Ability to explain judgments in two to four sentences.
• Careful reading to notice omissions in responses.
• Comfort with applying a detailed rubric consistently.
• Reliable availability for time-boxed batches.

Responsibilities

• Evaluate the relevance and clarity of AI responses.
• Detect sycophancy in AI interactions and identify when the AI tells users what they want to hear instead of what they need.
• Assess the tone and outcome of conversations to determine if users leave informed and calm or anxious and uninformed.
• Provide written justification for ratings below the top option with improvements.

Skills

Strong written English

Explain judgments succinctly

Careful reading

Rubric-based evaluation

Time management

Job description

About the job

Mercor connects elite creative and technical talent with leading AI research labs. Headquartered in San Francisco, our investors include Benchmark, General Catalyst, Peter Thiel, Adam D'Angelo, Larry Summers, and Jack Dorsey.

Position: Generalist Annotator — Health AI Conversation Quality Evaluation
Type: Contract
Compensation: $20–$160/hour
Location: Remote

Role Responsibilities

• Evaluate the relevance and clarity of AI responses. Ensure the AI answers the question in understandable language for non-experts.
• Detect sycophancy in AI interactions. Identify when the AI tells users what they want to hear instead of what they need.
• Assess the tone and outcome of conversations. Determine if the user leaves informed and calm or anxious and uninformed.
• Provide written justification for ratings below the top option. Explain what went wrong and suggest improvements.
• Work independently to review conversations ranging from 1 to 7 turns. Spend 2–13 minutes per conversation based on length.

Qualifications

Must-Have

• Strong written English skills.
• Ability to explain judgments in two to four specific sentences.
• Careful reading to notice omissions in responses.
• Comfort with applying a detailed rubric consistently.
• Reliable availability for time-boxed batches.

Preferred

• Experience in annotation, human-feedback, evaluation, or content review.
• Background in explaining technical concepts to non-experts.
• Experience with rubric-based grading or quality assurance.

Important

• Using an AI tool to write comments is prohibited. This work captures human judgment that AI lacks. AI-written feedback corrupts the dataset and is checked for.

Resources & Support

• For details about the interview process and platform information, please check: https://talent.docs.mercor.com/welcome
• For any help or support, reach out to: [e-mail]