LLM Annotator - Master's Degree · Turing Turing · Varies · remote · 2w ago Varies 2w ago

Benture; San Francisco (CA); Remote; USD 827,000 - 3,306,000; Part time

Job summary

Turing is seeking detail-oriented LLM Annotators for a remote, short-term contract role. You will evaluate and improve cutting-edge language models, design prompts, and validate outputs against source data.

Ideal candidates hold a Master's degree and have strong analytical and communication skills. The position offers flexible hours (20, 30, or 40 per week) with overlap for collaboration, and no medical or paid leave benefits as a contractor role.

Qualifications

• Master's degree or higher in any discipline.
• Minimum 3 years of professional, research, or teaching experience.
• Excellent written English communication skills.
• Exceptional attention to detail and ability to validate information against source data.

Responsibilities

• Design challenging prompts that evaluate an LLM's ability to retrieve, analyze, and reason over structured data.
• Assess AI-generated responses for factual accuracy, logical reasoning, and completeness.
• Identify model failures, hallucinations, inconsistencies, and reasoning gaps.
• Validate model outputs using provided datasets and supporting evidence.
• Document findings with clear, evidence-based explanations.
• Adhere consistently to annotation guidelines and maintain high-quality standards.

Skills

Analytical thinking

Attention to detail

Written English

Education

Master's degree or higher

Tools

CSV

Excel

Databases

Job description

LLM Annotator (Master's Required) | Contractor | Remote | 20–40 hrs/week

Turing is seeking detail-oriented LLM Annotators to help evaluate and improve cutting-edge Large Language Models. This is a short-term remote contract role ideal for analytical professionals with advanced degrees who enjoy working at the frontier of AI research.

About Turing

Based in San Francisco, Turing is the world's leading research accelerator for frontier AI labs and a trusted partner for global enterprises deploying advanced AI systems. Turing helps accelerate frontier research with high-quality data, advanced training pipelines, and top AI researchers specializing in coding, reasoning, STEM, multilinguality, multimodality, and agents.

Key Responsibilities

• Design challenging prompts that evaluate an LLM's ability to retrieve, analyze, and reason over structured data.
• Assess AI-generated responses for factual accuracy, logical reasoning, and completeness.
• Identify model failures, hallucinations, inconsistencies, and reasoning gaps.
• Validate model outputs using provided datasets and supporting evidence.
• Document findings with clear, evidence-based explanations.
• Adhere consistently to annotation guidelines and maintain high-quality standards.

Minimum Qualifications

• Master's degree or higher in any discipline.
• Minimum 3 years of professional, research, or teaching experience.
• Strong analytical and critical thinking skills.
• Excellent written English communication skills.
• Exceptional attention to detail and ability to validate information against source data.

Preferred Qualifications

• Experience working with Large Language Models (LLMs) or Generative AI.
• Familiarity with prompt engineering, AI evaluation, data annotation, or model testing.
• Experience with structured datasets (CSV, Excel, databases, etc.).
• Ability to identify edge cases and design prompts that expose model limitations.

Contract Details

• Commitment: 20, 30, or 40 hours per week with at least 4 hours PST overlap required.
• Employment Type: Contractor assignment (no medical/paid leave benefits).
• Duration: 4-week contract.

Why Join Turing?

• Work on cutting-edge AI research projects with real-world impact.
• Competitive compensation with flexible, remote working hours.
• Collaborate with leading AI researchers and engineers globally.