LLM Evaluation Specialist – Cultural and Linguistic Alignment - Arabic native speaker -

Innodata; Dammam; On-site; SAR 149,000 - 225,000; Full time

Job summary

A leading technology firm is seeking linguistically and culturally aware professionals to evaluate and enhance multilingual prompt-response datasets for large language models. The role involves rubric design, translation evaluation, and prompt creation. Candidates should possess native proficiency in Arabic and a Master's degree, with a preference for those experienced in LLM evaluation and prompt engineering. This position offers a unique opportunity to contribute to the cultural accuracy of LLM outputs.

Qualifications

• Deep familiarity with cultural norms in the corresponding region.
• Experience in content moderation or linguistic QA preferred.
• Comfortable working in spreadsheets and evaluation templates.

Responsibilities

• Update rubric definitions with Arabic specific examples.
• Identify need for additional rubrics tailored to specific languages.
• Review prompts translated from English to Arabic for accuracy.
• Rate prompt-response pairs using standardized evaluation templates.
• Document problematic outputs with clear explanations.

Skills

Native proficiency in Arabic

Attention to detail

Cultural awareness

Experience in LLM evaluation

Education

Master’s degree in relevant stream

Tools

Spreadsheets

Evaluation templates

Job description

Overview

We are looking for linguistically and culturally aware professionals to support the evaluation and enhancement of multilingual prompt-response datasets for large language models (LLMs). This role involves rubric design, evaluation of translations and model outputs, prompt creation, and red teaming focused on identifying and surfacing cultural nuances and biases in LLM behaviour.

Responsibilities

Key Responsibilities :

Rubric Definition & Prompt Evaluation

• Update rubric definitions with Arabic specific examples to ensure cultural and linguistic relevance.
• Identify the need for additional rubrics tailored to specific languages or regional contexts.
• Review prompts translated from English into Arabic and revise where translations appear unnatural or inaccurate.
• Writing of thoughtful prompts which can test the cultural awareness of LLM models.
• Rate prompt-response pairs using a standardized evaluation template based on rubrics and provide detailed justifications to base the findings.
• Document problematic outputs and annotate them with clear explanations of rubric violations or cultural insensitivities.

Required Qualifications

• Native proficiency in the Arabic and deep familiarity with cultural norms in the corresponding region.
• Experience in LLM evaluation, content moderation, or linguistic QA preferred.
• Strong attention to detail with the ability to identify subtle issues in language use, tone, and cultural references.
• Comfortable working in spreadsheets and evaluation templates.
• Master’s degree in relevant stream.

Preferred Qualifications

• Prior experience with prompt engineering or LLM testing.
• Familiarity with tools such as Gemini, ChatGPT or similar LLM platforms.
• Ability to clearly articulate reasoning behind rubric ratings or prompt edits.