Remote LLM Evaluation Annotator — Prompt Design & Validation

OpenTrain AI, Inc.; United States; Remote; USD 55,000 - 103,000; Full time

Job summary

OpenTrain AI, Inc. seeks contributors to test and improve large language models by evaluating how they retrieve, analyze, and reason over structured data. You will review generated answers for accuracy, logic, completeness, and evidence.

This remote contractor assignment lasts four weeks with 20–40 hours per week. At least four hours of overlap with Pacific Time is required. A master's degree or higher and at least three years of professional, research, or teaching experience are required.

Qualifications

• A master's degree or higher is required.
• At least three years of professional, research, or teaching experience is required.
• Excellent written English communication skills and close attention to detail are required.

Responsibilities

• Design challenging prompts that test model reasoning over datasets.
• Evaluate AI-generated answers for factual accuracy, logical reasoning, and completeness.
• Identify hallucinations, inconsistencies, reasoning gaps, edge cases, and other model failures.
• Validate responses against provided datasets and supporting evidence.
• Document clear, evidence-backed explanations while following annotation guidelines.

Skills

Analytical thinking

Critical thinking

Written English proficiency

Detail-oriented

Education

Master's degree or higher

Tools

CSV files

Spreadsheets

Databases

AI evaluation tools

Job description

OpenTrain AI, Inc. seeks contributors to test and improve large language models by evaluating how they retrieve, analyze, and reason over structured data. You will review generated answers for accuracy, logic, completeness, and evidence.

This remote contractor assignment lasts four weeks with 20–40 hours per week. At least four hours of overlap with Pacific Time is required. A master's degree or higher and at least three years of professional, research, or teaching experience are required.