LLM Response Evaluation Annotator

OpenTrain AI, Inc.; United States; Remote; USD 55,000 - 103,000; Full time

Job summary

OpenTrain AI, Inc. seeks contributors to test and improve large language models by evaluating how they retrieve, analyze, and reason over structured data. You will review generated answers for accuracy, logic, completeness, and evidence.

This remote contractor assignment lasts four weeks with 20–40 hours per week. At least four hours of overlap with Pacific Time is required. A master's degree or higher and at least three years of professional, research, or teaching experience are required.

Qualifications

• A master's degree or higher is required.
• At least three years of professional, research, or teaching experience is required.
• Excellent written English communication skills and close attention to detail are required.

Responsibilities

• Design challenging prompts that test model reasoning over datasets.
• Evaluate AI-generated answers for factual accuracy, logical reasoning, and completeness.
• Identify hallucinations, inconsistencies, reasoning gaps, edge cases, and other model failures.
• Validate responses against provided datasets and supporting evidence.
• Document clear, evidence-backed explanations while following annotation guidelines.

Skills

Analytical thinking

Critical thinking

Written English proficiency

Detail-oriented

Education

Master's degree or higher

Tools

CSV files

Spreadsheets

Databases

AI evaluation tools

Job description

The Work

You will help assess and improve large language models by testing how well they retrieve, analyze, and reason over structured data. You will review generated answers and explain whether they are accurate, logical, complete, and supported by evidence.

• Design challenging prompts that test model reasoning over datasets.
• Evaluate AI-generated answers for factual accuracy, logical reasoning, and completeness.
• Identify hallucinations, inconsistencies, reasoning gaps, edge cases, and other model failures.
• Validate responses against provided datasets and supporting evidence.
• Document clear, evidence-backed explanations while following annotation guidelines and consistent quality standards.

What It Pays And Takes

The role is a remote contractor assignment lasting four weeks. Pay details are not provided in the listing.

• Remote assignment open worldwide.
• Available commitments: 20, 30, or 40 hours per week.
• At least four hours of overlap with Pacific Time is required.
• A master's degree or higher in any discipline is required.
• At least three years of professional, research, or teaching experience is required.
• Strong analytical and critical-thinking skills are required.
• Excellent written English communication skills and close attention to detail are required.
• You must be able to validate information against source data.
• Experience with large language models, generative AI, prompt engineering, AI evaluation, data annotation, model testing, CSV files, spreadsheets, or databases is helpful.

About AI Training Work

AI training is the human work behind systems that generate and understand text. People write prompts, review model responses, and give detailed feedback so these systems become more accurate, useful, and reliable. Experienced contributors are valued for careful judgment and subject knowledge.