Remote AI Quality Evaluator & Feedback Specialist

YO AI Labs; Sydney; Remote; AUD 83,000 - 138,000; Full time

Job summary

YO AI Labs is seeking an AI Evaluation Specialist to assess outputs from enterprise AI training projects. This contractor role is remote and can be done from Australia, the US, UK, and other regions, with flexible hours to cover high-volume evaluation tasks.

You will apply rubrics to judge accuracy and relevance, identify gaps, and provide concise feedback to improve AI systems. Strong writing and detail orientation are essential for consistent, fair assessments.

Qualifications

• Experience in grading, QA, editorial review or similar analytical work.
• Daily use of AI assistants such as ChatGPT or Claude.
• Strong written communication and ability to summarize findings.
• Experience with rubric development or quality assessment.
• Attention to detail and consistency in evaluations.

Responsibilities

• Evaluate AI outputs against rubrics for accuracy, relevance, quality, and guidelines.
• Apply consistent judgment across large volumes of examples.
• Identify reasoning gaps, tool-use issues, and inconsistencies.
• Provide concise, actionable feedback for improvement.
• Document evaluations and recommendations accurately.
• Participate in rubric interpretation discussions and process improvements.

Skills

Quality assurance

Editorial review

Annotation

Analytical thinking

Tools

ChatGPT

Claude

Job description

YO AI Labs is seeking an AI Evaluation Specialist to assess outputs from enterprise AI training projects. This contractor role is remote and can be done from Australia, the US, UK, and other regions, with flexible hours to cover high-volume evaluation tasks.

You will apply rubrics to judge accuracy and relevance, identify gaps, and provide concise feedback to improve AI systems. Strong writing and detail orientation are essential for consistent, fair assessments.