AI Evaluation Guidelines Architect

24-Mag Llc

Northern (KY)

Hybrid

USD 55,000 - 83,000

Full time

5 days ago

Be an early applicant

Application generator

An application made for this job — a tailored resume and cover letter that speak straight to the posting.

Get past ATS filters

Job summary

24-MAG LLC is seeking experienced linguists, instructional designers, and technical writers for a remote, full-time role to develop rater guidelines and scoring rubrics for generative AI evaluation. You will translate complex requirements into actionable instructions for human evaluators across finance, retail, insurance, legal, sports, and more.

Ideal candidates have 3+ years in related fields and a track record of reducing ambiguity with clear guidance.

Qualifications

3+ years of professional experience in linguistics, instructional design, technical writing, or related field.
Experience developing guidelines and rubrics for human evaluators in generative AI or RLHF projects.
Ability to resolve ambiguity and contradictions in complex specifications.
Translate specialist requirements into clear, practical instructions.
Ability to work across multiple domains with SME collaboration.

Responsibilities

Rater guideline development: translate requirements into clear, structured instructions.
Rubric & evaluation framework design with measurable criteria.
Ambiguity and consistency review to reduce contradictions.
Cross-domain instructional translation for finance, retail, insurance, legal, and more.