Remote AI Evaluation Guidelines Architect

TrulyHired; United States; On-site; USD 61,992 - 89,544; Full time

Benefits offered by this job

Fully remote engagement

Job summary

TrulyHired is seeking an AI Rater Guidelines Writer (Linguist / Instructional Designer) for a fully remote, independent-contract engagement. The role focuses on turning complex program requirements into clear, structured rater guidelines for AI data evaluation, with a commitment of about 35 hours per week.

You will design evaluation rubrics, review existing documentation for ambiguity, and collaborate with researchers and subject matter experts to ensure consistency across guideline sets used in

Qualifications

• Minimum 3 years of professional experience in linguistics, instructional design, technical writing, or AI content development.
• Experience creating or refining evaluation guidelines, rubrics, or reviewer instructions in Generative AI environments.
• Ability to convert domain knowledge into clear, structured documentation.

Responsibilities

• Translate complex program requirements into clear, concise rater guidelines.
• Design evaluation rubrics and scoring frameworks for consistent AI task evaluation.
• Review documentation to identify ambiguity and gaps, revising guidelines for clarity.

Skills

Attention to detail

Written communication

Analytical thinking

Job description

TrulyHired is seeking an AI Rater Guidelines Writer (Linguist / Instructional Designer) for a fully remote, independent-contract engagement. The role focuses on turning complex program requirements into clear, structured rater guidelines for AI data evaluation, with a commitment of about 35 hours per week.

You will design evaluation rubrics, review existing documentation for ambiguity, and collaborate with researchers and subject matter experts to ensure consistency across guideline sets used in