Remote LLM Evaluator & Prompt Engineer (Contract)

Benture; San Francisco (CA); Remote; USD 827,000 - 3,306,000; Part time

Job summary

Turing is seeking detail-oriented LLM Annotators for a remote, short-term contract role. You will evaluate and improve cutting-edge language models, design prompts, and validate outputs against source data.

Ideal candidates hold a Master's degree and have strong analytical and communication skills. The position offers flexible hours (20, 30, or 40 per week) with overlap for collaboration, and no medical or paid leave benefits as a contractor role.

Qualifications

• Master's degree or higher in any discipline.
• Minimum 3 years of professional, research, or teaching experience.
• Excellent written English communication skills.
• Exceptional attention to detail and ability to validate information against source data.

Responsibilities

• Design challenging prompts that evaluate an LLM's ability to retrieve, analyze, and reason over structured data.
• Assess AI-generated responses for factual accuracy, logical reasoning, and completeness.
• Identify model failures, hallucinations, inconsistencies, and reasoning gaps.
• Validate model outputs using provided datasets and supporting evidence.
• Document findings with clear, evidence-based explanations.
• Adhere consistently to annotation guidelines and maintain high-quality standards.

Skills

Analytical thinking

Attention to detail

Written English

Education

Master's degree or higher

Tools

CSV

Excel

Databases

Job description

Turing is seeking detail-oriented LLM Annotators for a remote, short-term contract role. You will evaluate and improve cutting-edge language models, design prompts, and validate outputs against source data.

Ideal candidates hold a Master's degree and have strong analytical and communication skills. The position offers flexible hours (20, 30, or 40 per week) with overlap for collaboration, and no medical or paid leave benefits as a contractor role.