AI Prompt Quality Rater

AI Trainer Jobs; United States; À distance; USD 48 000 - 76 000; Plein temps

Résumé du poste

AI Trainer Jobs is seeking a detail-oriented evaluator to score prompt-response pairs from large language models. You will use structured rubrics to rate factual accuracy, instruction-following, and coherence across batches, typically 20–50 pairs per session.

You will flag edge cases, document subtle failure modes, and contribute to fine-tuning data. Sessions are asynchronous with calibration meetings every two weeks, offering potential tracks in legal, medical, or coding domains.

Qualifications

• Experience evaluating prompts using structured rubrics.
• Ability to identify edge cases and subtle failure modes.
• Strong analytical writing to justify scores.

Responsabilités

• Evaluate prompts and model responses using a rubric.
• Identify and flag edge cases and misleading outputs.
• Write concise justifications for non-obvious scores and share findings.

Connaissances

Prompt evaluation

Rubric-based scoring

LLM output analysis

Instruction-following assessment

Factual accuracy verification

Annotation platform proficiency

Description du poste

Pay: $35-55/hour

You'll evaluate prompt-response pairs generated by large language models, scoring them across dimensions like factual accuracy, instruction-following, coherence, and appropriate refusal behavior. Each session involves reviewing batches of 20-50 pairs using a structured rubric inside a web-based annotation platform, flagging edge cases, and writing brief justifications for non-obvious scores.

Beyond surface-level ratings, you'll identify subtle failure modes — responses that are technically accurate but misleading, answers that follow the letter of a prompt while violating its intent, and outputs that pass a quick read but contain embedded errors. You'll work asynchronously, with a typical batch taking 60-90 minutes, and you're expected to maintain inter-annotator agreement scores above 0.75 kappa.

Feedback you submit feeds directly into preference datasets used to fine-tune and align production models. Calibration sessions run bi-weekly, where you'll review disagreements with a senior evaluator and update your scoring approach. Strong performance can lead to specialized tracks covering domain-specific evaluation (legal, medical, code).

Skills & requirements

• Prompt evaluation
• Rubric-based scoring
• LLM output analysis
• Instruction-following assessment
• Factual accuracy verification
• Annotation platform proficiency

Category: RLHF