Health AI Conversation Quality Evaluator (Remote, Contract)

Mercor; San Francisco (CA); Hybrid; GBP 21,000 - 166,000; Part time

Job summary

Mercor is seeking a Generalist Annotator for Health AI Conversation Quality Evaluation. This remote contract role pays $20–$160/hour. You will independently review conversations (1–7 turns) and provide concise, rubric-based feedback to improve AI interactions.

Strong written English and the ability to explain judgments in 2–4 sentences are required. Ability to read carefully and adhere to a detailed rubric is essential. Remote work is offered in the UK and beyond.

Qualifications

• Strong written English required.
• Ability to explain judgments in two to four sentences.
• Careful reading to notice omissions in responses.
• Comfort with applying a detailed rubric consistently.
• Reliable availability for time-boxed batches.

Responsibilities

• Evaluate the relevance and clarity of AI responses.
• Detect sycophancy in AI interactions and identify when the AI tells users what they want to hear instead of what they need.
• Assess the tone and outcome of conversations to determine if users leave informed and calm or anxious and uninformed.
• Provide written justification for ratings below the top option with improvements.

Skills

Strong written English

Explain judgments succinctly

Careful reading

Rubric-based evaluation

Time management

Job description

Mercor is seeking a Generalist Annotator for Health AI Conversation Quality Evaluation. This remote contract role pays $20–$160/hour. You will independently review conversations (1–7 turns) and provide concise, rubric-based feedback to improve AI interactions.

Strong written English and the ability to explain judgments in 2–4 sentences are required. Ability to read carefully and adhere to a detailed rubric is essential. Remote work is offered in the UK and beyond.