AI Output Quality Evaluator - Remote Contractor

YO AI Labs; Riyadh; Remote; SAR 129,000 - 232,000; Full time

Job summary

YO AI Labs is seeking an AI Evaluation Specialist to assess and improve the quality of AI assistant outputs for an enterprise training project. This contractor role is remote with responsibilities across multiple regions.

You will apply rubrics to measure accuracy, relevance, and adherence to guidelines, provide actionable feedback, and help refine evaluation criteria through ongoing collaboration.

Qualifications

• Experience in grading, QA, editorial review or similar analytical roles.
• Daily use of AI assistants like ChatGPT or Claude.
• Strong written communication of findings.
• Experience with rubric development or quality assessment.
• Ability to maintain consistency, fairness and attention to detail.

Responsibilities

• Evaluate AI outputs against rubrics for accuracy, relevance, quality, and guidelines.
• Apply objective judgment across large volumes of examples.
• Identify reasoning gaps, tool-use issues, and inconsistencies.
• Provide concise, actionable feedback to improve AI systems.
• Document evaluations and recommendations accurately.
• Participate in rubric interpretation discussions and evolve standards.
• Contribute to process improvements and best practices.
• Maintain thorough evaluation records.

Skills

Quality assurance

Editorial review

Annotation

AI tools

Analytical thinking

Process improvement

Independent work

Attention to detail

Written communication

Tools

ChatGPT

Claude

Job description

YO AI Labs is seeking an AI Evaluation Specialist to assess and improve the quality of AI assistant outputs for an enterprise training project. This contractor role is remote with responsibilities across multiple regions.

You will apply rubrics to measure accuracy, relevance, and adherence to guidelines, provide actionable feedback, and help refine evaluation criteria through ongoing collaboration.