Arabic LLM Evaluator & Data Annotation Specialist (Contract)

Mercor; Dubai; On-site; AED 101,000 - 132,000; Full time

Job summary

Mercor is seeking a Language Model Evaluator to join our team. You will evaluate LLM-generated responses, conduct fact-checking using trusted sources and external tools, and generate high-quality evaluation data with clear annotations.

Requirements include a bachelor's degree, native Arabic with CEFR C2, strong writing, attention to detail, and comfort moving across topics. This is a contract role (full-time or part-time) paying $23/hour, geographically restricted to Egypt, Saudi Arabia, UAE,

Qualifications

• Bachelor's degree required
• Native speaker or ILR 5/CEFR C2 in Arabic
• Significant experience using large language models (LLMs)
• Excellent writing skills
• Strong attention to detail
• Adaptable and comfortable moving across topics and customer requirements
• Background in domains requiring structured analytical thinking
• Excellent college-level mathematics skills

Responsibilities

• Evaluate LLM-generated responses for accuracy, usefulness, and alignment with guidelines.
• Conduct fact-checking using trusted public sources and external tools.
• Generate high-quality human evaluation data with annotated strengths and weaknesses.
• Assess reasoning quality, clarity, tone, and completeness of responses.
• Ensure model outputs align with expected conversational behavior and system guidelines.
• Apply consistent annotations following defined rubrics and benchmarks.

Skills

LLM experience

Attention to detail

Writing skills

Analytical thinking

Mathematics

Adaptability

Domain knowledge

Education

Bachelor's degree

Tools

External tools

Job description

Mercor is seeking a Language Model Evaluator to join our team. You will evaluate LLM-generated responses, conduct fact-checking using trusted sources and external tools, and generate high-quality evaluation data with clear annotations.

Requirements include a bachelor's degree, native Arabic with CEFR C2, strong writing, attention to detail, and comfort moving across topics. This is a contract role (full-time or part-time) paying $23/hour, geographically restricted to Egypt, Saudi Arabia, UAE,