Remote Language Model Evaluator (Arabic, Native Speaker)

Mercor; Dubai; Remote; AED 96,000 - 137,000; Full time

Job summary

Mercor is seeking a Language Model Evaluator to assess LLM responses for accuracy, relevance, and safety. You will verify facts, use external tools, and generate detailed evaluation data.

The role requires native Arabic fluency (CEFR C2), a bachelor’s degree, and proven experience with large language models. Strong writing and analytical skills are essential for success in this contract role.

Qualifications

• Bachelor's degree required.
• Native speaker or ILR 5/CEFR C2 in Arabic required.
• Significant experience using large language models (LLMs).
• Excellent writing skills required.
• Strong attention to detail.
• Adaptable across topics, domains and customer requirements.
• Background in structured analytical thinking and college-level mathematics.

Responsibilities

• Evaluate LLM-generated responses for quality and correctness.
• Conduct fact-checking using trusted sources and external tools.
• Annotate responses to create high-quality evaluation data.
• Assess reasoning quality, clarity, tone and completeness.
• Ensure model outputs align with guidelines and expectations.
• Apply consistent annotations using defined rubrics and benchmarks.

Skills

Arabic (CEFR C2)

LLM experience

Excellent writing

Attention to detail

Adaptable across topics

Structured analytical thinking

Mathematics skills

Education

Bachelor's degree

Tools

RLHF evaluation tools

Job description

Mercor is seeking a Language Model Evaluator to assess LLM responses for accuracy, relevance, and safety. You will verify facts, use external tools, and generate detailed evaluation data.

The role requires native Arabic fluency (CEFR C2), a bachelor’s degree, and proven experience with large language models. Strong writing and analytical skills are essential for success in this contract role.