Instruction Following Data Labeler

AI Trainer Jobs; United States; Remote; USD 28,000 - 48,000; Full time

Job summary

AI Trainer Jobs seeks detail-oriented labeling specialists to work on instruction-following data quality projects. You will label text data, assess model outputs against constraints, and apply rubric-based scoring across tasks.

Files are processed in batches with asynchronous workflows. A throughput target of 30–50 tasks per hour is typical; strong reading skills and adherence to guidelines are essential. Prior data-labeling or moderation experience is a plus.

Qualifications

• Strong reader with high attention to detail.
• Experience in content moderation, linguistic annotation, or data quality roles is transferable.
• Read instructions carefully and flag ambiguous cases.

Responsibilities

• Label text data for instruction-following evaluation.
• Apply rubric-based scoring across tasks and provide span-level annotations.
• Work with batch-style, asynchronous labeling workflows.

Skills

Instruction interpretation

Rubric-based labeling

Span-level annotation

Constraint adherence

Attention to detail

Guideline compliance

Job description

Pay: $20–35/hour

You'll label text data focused on instruction-following behavior — specifically, evaluating whether a model's output correctly satisfies the constraints and intent of a given user instruction. Tasks include binary labels (did the model follow the instruction — yes/no), multi-attribute scoring (tone, format, completeness, constraint adherence), and span-level annotation (highlight exactly which part of the output fails to follow a stated constraint). You'll work with a detailed labeling guide and per-project instructions that define edge cases explicitly.

The data covers a wide range of instruction types: formatting requests ("respond only in bullet points"), content constraints ("do not mention competitors"), persona instructions, length targets, and structured output requirements (JSON, tables, lists). Each task is self-contained — you read the instruction, read the output, apply the rubric, and submit. Throughput targets are typically 30–50 tasks per hour for straightforward binary tasks, lower for multi-attribute annotation. Work is async and batched.

You should be a strong reader with high attention to detail and the ability to apply written guidelines consistently across varied examples. Prior experience in content moderation, linguistic annotation, or data quality work is directly transferable. You don't need a technical background — but you do need to read instructions carefully and flag ambiguous cases through the proper escalation channel rather than guessing.

Skills & requirements

• Instruction interpretation
• Rubric-based text labeling
• Span-level annotation
• Constraint adherence evaluation
• Attention to detail
• Annotation guideline compliance

Category: Data Labeling