Remote AI Evaluation Specialist (Contractor)

YO AI Labs; Valencia; A distancia; EUR 53.000 - 80.000; Jornada completa

Descripción de la vacante

YO AI Labs is seeking an AI Evaluation Specialist to join as a remote contractor. You will assess AI-generated responses against detailed rubrics, identify issues, and provide clear, actionable feedback to raise overall AI quality.

The role emphasizes consistency, fairness, and the ability to work independently on large evaluation tasks while contributing to rubric discussions and process improvements.

Formación

• Experience in grading, QA, editorial review or similar analytical work.
• Advanced daily use of AI assistants like ChatGPT, Claude or similar tools.
• Strong ability to analyze complex information and communicate findings in writing.
• Experience with process improvement, rubric development, or quality assessment.
• Strong critical-thinking with focus on consistency, accuracy, and fairness.
• Comfortable working independently on high-volume evaluation tasks.
• Strong attention to detail and ability to identify subtle quality issues.
• Collaborative approach to resolving ambiguous cases and improving criteria.

Responsabilidades

• Evaluate AI-generated outputs against rubrics for accuracy, relevance, quality, and guidelines.
• Apply consistent and objective judgment across large volumes of examples.
• Identify reasoning gaps, tool-use issues, and inconsistencies.
• Provide concise, actionable feedback to improve AI systems.
• Participate in rubric interpretation discussions and evolving quality standards.
• Contribute to process improvements and evaluation best practices.
• Maintain accurate documentation of evaluations and recommendations.

Conocimientos

Quality assurance

Editorial review

Analytical writing

Rubric development

Critical thinking

Independent evaluation

Attention to detail

Collaboration

Herramientas

AI tools (ChatGPT/Claude)

Descripción del empleo

YO AI Labs is seeking an AI Evaluation Specialist to join as a remote contractor. You will assess AI-generated responses against detailed rubrics, identify issues, and provide clear, actionable feedback to raise overall AI quality.

The role emphasizes consistency, fairness, and the ability to work independently on large evaluation tasks while contributing to rubric discussions and process improvements.