AI Evaluation Guidelines & Rubric Specialist

ixolabs.ai; France; Hybride; EUR 60 000 - 84 000; Plein temps

Résumé du poste

IXO is seeking a seasoned reviewer and guideline engineer to translate complex program requirements into explicit rating criteria and edge-case examples.

You will formalize decision rules for AI outputs, resolve ambiguities, and produce revision rationales while maintaining consistency across related guideline sets. US-based contributors are preferred; about 35 hours per week with flexible scheduling.

Qualifications

• Three+ years of professional experience in linguistics, instructional design, technical writing or related field.
• Direct work on human-rater guidelines or rubrics in generative AI or RLHF.
• Ability to explain nuanced instructions clearly and resolve ambiguities.

Responsabilités

• Translate program requirements into explicit rating criteria and decision rules.
• Create edge-case examples and rating rubrics.
• Identify contradictions, vague terms and missing cases in guidelines.
• Test guidelines against difficult outputs and refine for clarity.
• Explain domain-specific distinctions for reviewers with different backgrounds.
• Produce documented changes showing revision improvements.

Connaissances

Linguistics

Instructional Design

Technical Writing

RLHF Guidelines

Description du poste

Your work at IXO

Write the instructions that allow different human reviewers to judge the same AI output consistently. You will turn incomplete or ambiguous specifications into practical guidance, identify where a rubric fails and refine the document until a reviewer can apply it with a defensible explanation.

Responsibilities

• Translate program requirements into explicit rating criteria, decision rules and examples.
• Find contradictions, vague terms and missing cases in draft guidelines.
• Test how the instructions handle difficult or borderline outputs, then revise the guidance to reduce unnecessary escalation.
• Explain domain-specific distinctions without assuming every rater has the same specialist background.
• Reconcile feedback from subject experts and maintain consistent rules across related guideline sets.
• Produce documented changes that show how a revision improves clarity and agreement.

Qualifications

• You need at least three years of professional experience in linguistics, instructional design, technical writing or a closely related field, including direct work on human-rater guidelines or rubrics in generative AI or RLHF.
• Bring concrete examples of resolving ambiguity or contradiction in written specifications, a record of increasing professional responsibility and the ability to explain nuanced instructions clearly. You should have demonstrated work across multiple subject areas and be available for at least 35 weekday hours per week for the intended work pattern.

Deliverables and terms

Deliverables include usable guideline sets, rating rubrics, edge-case examples and revision rationales. The opportunity is for US-based contributors. IXO will agree the assignment scope and schedule before work starts.

Pay

Pay is $50 $70/hr (USD).