Teacher | $36.16/hr Remote

Crossing Hurdles; Canada; On-site; CAD 60,000 - 80,000; Full time

Job summary

A leading tech evaluation company in Canada seeks an AI Evaluation Specialist to assess LLM-generated responses for accuracy and relevance. The ideal candidate will be fluent in French and proficient in English, with a background in analytical fields. Responsibilities include fact-checking, annotating AI response strengths and weaknesses, and ensuring adherence to guidelines. This role requires strong writing skills and attention to detail, making it suitable for candidates with experience in research or analytics.

Qualifications

• Fluent in French (ILR 5 / CEFR C2) and proficient in English.
• Experience with LLMs and understanding of their practical applications.
• Ability to provide structured feedback on AI outputs.
• Ability to work across varied topics and evolving requirements.

Responsibilities

• Evaluate LLM-generated responses for accuracy and relevance.
• Perform fact-checking using reliable sources.
• Create quality evaluation data by annotating AI responses.
• Assess reasoning quality and adherence to guidelines.

Skills

Native-level French fluency

Strong English proficiency

Experience with large language models

Excellent writing skills

Strong attention to detail

Analytical thinking

Education

Background in research, policy, analytics, linguistics, or engineering

Job description

• Evaluate LLM-generated responses for accuracy, relevance, and effectiveness across a wide range of topics.
• Perform fact-checking using reliable public sources and external tools.
• Create high-quality human evaluation data by annotating response strengths, gaps, and factual errors.
• Assess reasoning quality, tone, clarity, and completeness of AI-generated outputs.
• Ensure responses follow expected conversational behavior and system guidelines.
• Apply consistent annotations using defined taxonomies, benchmarks, and evaluation frameworks.

Requirements

• Native-level or near-native fluency in French (ILR 5 / CEFR C2) with strong English proficiency.
• Proven experience using large language models and understanding real-world LLM use cases.
• Excellent writing skills with the ability to provide clear, nuanced, and structured feedback.
• Strong attention to detail and analytical thinking.
• Ability to work across multiple topics, domains, and evolving requirements.
• Background in structured analytical fields such as research, policy, analytics, linguistics, or engineering.