Remote Native-Language Benchmark Engineer (Freelance)

LILT (Production); Austria; Remote; EUR 55,000 - 103,000; Full time

Benefits offered by this job

Flexible schedule

Remote work

Job summary

Lilt is seeking experienced native-speaking software engineers to design, build, and validate multilingual benchmarks. This remote, freelance role focuses on creating high-signal tasks in native languages and evaluating multilingual robustness without English translation crutches.

You’ll work on task engineering, asset creation, prompting and translation, verification scripts, and calibration across model tiers, within a rigorous 4‑layer quality control process.

Qualifications

• 1+ years of industry software engineering experience.
• Native or near-native fluency in the target language with strong grammar and phrasing.
• High English proficiency.
• Strong Python, shell scripting and data processing skills.
• Experience with Terminal/CLI workflows and coding agents.
• Deep understanding of multilingual text processing pitfalls, including encoding/decoding robustness and Unicode normalization.

Responsibilities

• Design, build and validate multilingual benchmarks.
• Create assets in the target language without English translation.
• Develop prompts and translations to reveal failure points in native language.
• Write deterministic verifier scripts and reference implementations.
• Calibrate task difficulty across model tiers and run configurations.
• Participate in a 4-layer quality control process for fairness and accuracy.

Skills

Python

Shell scripting

Data processing

Terminal/CLI workflows

Multilingual text processing

Unicode handling

Regex & parsing

Tools

Unix/Linux

Git

Job description

Lilt is seeking experienced native-speaking software engineers to design, build, and validate multilingual benchmarks. This remote, freelance role focuses on creating high-signal tasks in native languages and evaluating multilingual robustness without English translation crutches.

You’ll work on task engineering, asset creation, prompting and translation, verification scripts, and calibration across model tiers, within a rigorous 4‑layer quality control process.