Remote AI Evaluation Guidelines Architect

Weekday AI; United States; À distance; USD 62 000 - 90 000; Plein temps

Résumé du poste

Weekday AI is hiring for a remote, full-time role focused on shaping how AI systems are trained through precise guidelines for human evaluators. You will translate complex requirements into clear, actionable rater instructions across finance, retail, insurance, legal, and sports.

Work is fully remote with a 35-hour weekday commitment. The position emphasizes documenting evaluation rubrics, improving clarity, and maintaining consistency across multiple domains in collaboration with researchers

Qualifications

• Minimum 3 years of professional experience in linguistics, instructional design, technical writing, or AI content development.
• Direct experience creating or refining evaluation guidelines, rating rubrics, or reviewer instructions in generative AI or AI data annotation.
• Ability to convert domain knowledge into clear, structured documentation across multiple subjects.
• Proven ability to resolve ambiguity and improve specifications with measurable improvements.
• Strong analytical thinking and exceptional attention to detail.
• Clear career progression and professional growth.
• Ability to commit 35+ hours per week during weekdays.
• Outstanding written communication to explain nuanced concepts clearly.

Responsabilités

• Translate complex program requirements into clear, concise, and easy-to-follow rater guidelines.
• Design comprehensive evaluation rubrics and scoring frameworks that enable consistent decision-making across a variety of AI tasks.
• Review existing documentation to identify ambiguity, inconsistencies, contradictions, and coverage gaps, then revise guidelines to improve clarity and usability.
• Convert subject-matter requirements from domains such as finance, retail, insurance, legal, and sports into structured evaluation instructions that non-domain raters can apply confidently.
• Collaborate with research teams, program managers, and subject matter experts to maintain consistency across multiple guideline sets.
• Develop documentation that addresses edge cases, exceptions, and complex evaluation scenarios while minimizing reviewer escalation.

Connaissances

Linguistics

Instructional Design

Technical Writing

AI Content Development

Description du poste

Weekday AI is hiring for a remote, full-time role focused on shaping how AI systems are trained through precise guidelines for human evaluators. You will translate complex requirements into clear, actionable rater instructions across finance, retail, insurance, legal, and sports.

Work is fully remote with a 35-hour weekday commitment. The position emphasizes documenting evaluation rubrics, improving clarity, and maintaining consistency across multiple domains in collaboration with researchers