Multimodal AI Session Evaluator

Appsierra Group; Australia; Remote; AUD 50,000 - 62,000; Full time

Job summary

Appsierra Group is hiring for an AI Session Annotator. This fully remote, independent contractor role focuses on evaluating AI assistants interpreting real-world video content, identifying inaccuracies, and providing precise feedback with evidence from sessions.

You will review turn-by-turn interactions, score based on a strict rubric, and document findings. Start date is immediate with flexible scheduling on project needs.

Qualifications

• : Native or native-level English proficiency, both spoken and written.
• Excellent written English and the ability to communicate observations clearly and precisely.
• Strong attention to detail and analytical judgment.
• Ability to follow a detailed evaluation rubric consistently.
• Ability to distinguish between what is actually visible in a video and what an AI assistant claims to see.
• An Android smartphone with a working camera and microphone — mandatory for this project.

Responsibilities

• Review recorded sessions where users interact with an AI assistant through voice and camera.
• Compare the assistant’s responses with the visual information actually available in the video.
• Identify fabricated, hallucinated, or inaccurate visual details.
• Evaluate each session for: Visual interpretation, Accuracy, Relevance, Response timing, and Appropriate use of spatial and directional language.
• Assess whether the assistant provides appropriate and timely warnings when responses could affect a user's health, safety, or financial security.
• Write detailed comments supporting each score with specific evidence from the session.
• Document conversations turn by turn using the provided intake form.
• Apply the evaluation rubric consistently without introducing personal scoring criteria.

Skills

Attention to detail

Analytical thinking

Strong written English

Follow rubric

Distinguish video vs AI claims

Tools

Android smartphone

Job description

Appsierra Group is hiring for an AI Session Annotator. This fully remote, independent contractor role focuses on evaluating AI assistants interpreting real-world video content, identifying inaccuracies, and providing precise feedback with evidence from sessions.

You will review turn-by-turn interactions, score based on a strict rubric, and document findings. Start date is immediate with flexible scheduling on project needs.