Remote Multilingual AI Benchmark Engineer

LILT; United States; Remote; USD 120,000 - 180,000; Full time

Job summary

LILT is seeking experienced software engineers to design, build, and validate multilingual benchmarks for evaluating language models. You will craft high-quality, native-language tasks and reference implementations, ensuring benchmarks truly test multilingual handling without English translation crutches.

The role emphasizes remote, freelance collaboration with a strong focus on encoding robustness, locale-sensitive processing, and rigorous QA to maintain benchmark integrity across diverse

Qualifications

• 5+ years of industry experience in software engineering.
• Proven track record at leading technology companies or top-tier universities.
• Native or near-native fluency in the target language with strong English proficiency.
• Strong Proficiency in Python, shell scripting, and data processing.
• Extensive experience with Terminal/CLI-based development workflows and coding agents.
• Deep understanding of multilingual text processing pitfalls (Encoding/Unicode, locale conventions, text I/O).

Responsibilities

• Design and build unit-under-test benchmarks and tasks for multilingual software challenges.
• Create high-signal assets in the native language to measure multilingual handling.
• Identify failure points in prompts and translations in the native language.
• Develop reference implementations and robust verifier scripts.
• Calibrate task difficulty across model tiers using standard Terminal-Bench configurations.
• Participate in a 4-layer QA process to ensure fairness and accuracy.

Skills

Python

Shell scripting

Data processing

Terminal/CLI workflows

Multilingual text processing

Unicode handling

Job description

LILT is seeking experienced software engineers to design, build, and validate multilingual benchmarks for evaluating language models. You will craft high-quality, native-language tasks and reference implementations, ensuring benchmarks truly test multilingual handling without English translation crutches.

The role emphasizes remote, freelance collaboration with a strong focus on encoding robustness, locale-sensitive processing, and rigorous QA to maintain benchmark integrity across diverse