Remote Arabic AI Benchmark Engineer - CLI & Multilingual Testing

Hidden Jobs; United States; Remote; USD 11,000 - 22,000; Part time

Job summary

Hidden Jobs is seeking a remote, freelance software engineer based in the United Arab Emirates to contribute to a rigorous benchmark for multilingual CLI tasks. The role emphasizes Arabic-language assets, Unicode and RTL handling, and evaluating coding agents across varied model sizes.

Ideal candidates will have 5+ years of software engineering experience, strong Python and shell scripting skills, and deep CLI development expertise with multilingual text processing knowledge.

Qualifications

• 5+ years of professional software engineering experience.
• Degree from a top-tier engineering university.
• Native or near-native Arabic fluency with high English proficiency.
• Strong Python, shell scripting, and data processing skills.
• Deep CLI development experience and familiarity with coding agents.
• Solid grasp of multilingual text processing issues, including Unicode normalization and RTL handling.

Responsibilities

• Engineer benchmark tasks designed to evaluate coding agents across realistic CLI scenarios.
• Construct task environments using Arabic-language assets that remain in the target language to genuinely test multilingual handling.
• Surface AI failure points through Arabic prompting and translation probing.
• Develop reference solutions and write reliable, deterministic verifier scripts, with rubric-based judging reserved for narrow cases.
• Analyze execution logs and tune task difficulty from Easy to Very Hard across multiple model sizes.
• Take part in a four-layer human quality assurance process alongside automated LLM-based checks to safeguard fairness and integrity.

Job description

Hidden Jobs is seeking a remote, freelance software engineer based in the United Arab Emirates to contribute to a rigorous benchmark for multilingual CLI tasks. The role emphasizes Arabic-language assets, Unicode and RTL handling, and evaluating coding agents across varied model sizes.

Ideal candidates will have 5+ years of software engineering experience, strong Python and shell scripting skills, and deep CLI development expertise with multilingual text processing knowledge.