AI Prompt Quality Evaluator & Feedback Analyst

AI Trainer Jobs; United States; Remote; USD 48,000 - 76,000; Full time

Job summary

AI Trainer Jobs is seeking a detail-oriented evaluator to score prompt-response pairs from large language models. You will use structured rubrics to rate factual accuracy, instruction-following, and coherence across batches, typically 20–50 pairs per session.

You will flag edge cases, document subtle failure modes, and contribute to fine-tuning data. Sessions are asynchronous with calibration meetings every two weeks, offering potential tracks in legal, medical, or coding domains.

Qualifications

• Experience evaluating prompts using structured rubrics.
• Ability to identify edge cases and subtle failure modes.
• Strong analytical writing to justify scores.

Responsibilities

• Evaluate prompts and model responses using a rubric.
• Identify and flag edge cases and misleading outputs.
• Write concise justifications for non-obvious scores and share findings.

Skills

Prompt evaluation

Rubric-based scoring

LLM output analysis

Instruction-following assessment

Factual accuracy verification

Annotation platform proficiency

Job description

AI Trainer Jobs is seeking a detail-oriented evaluator to score prompt-response pairs from large language models. You will use structured rubrics to rate factual accuracy, instruction-following, and coherence across batches, typically 20–50 pairs per session.

You will flag edge cases, document subtle failure modes, and contribute to fine-tuning data. Sessions are asynchronous with calibration meetings every two weeks, offering potential tracks in legal, medical, or coding domains.