Remote Native-Language AI Benchmark Engineer

LILT (Production); United States; Remote; USD 83,000 - 138,000; Full time

Benefits offered by this job

Remote work

Flexible schedule

Global collaboration

Job summary

LILT is seeking experienced software engineers to design, build, and validate a multilingual benchmark suite for evaluating language models. You will create high-signal tasks in your native language, ensuring assets remain in the target language to measure multilingual handling without English translation crutches.

As a remote freelance contractor, you’ll perform task engineering, asset creation, prompting and translation, and robust verification scripts, while calibrating task difficulty across

Qualifications

• 5+ years of industry software engineering experience.
• Native or near-native English proficiency with strong grammar and style.
• Strong Python, shell scripting, and data processing skills.
• Extensive experience with Terminal/CLI-based development workflows.
• Deep technical understanding of multilingual text processing pitfalls.

Responsibilities

• Task engineering: evaluating coding agents.
• Asset creation: build realistic task environments in native language.
• Prompting & translation: identify failure points in the native language.
• Implementation & verification: develop robust solutions and verifier scripts.
• Calibration & execution: analyze logs and calibrate task difficulty across model tiers.
• Quality assurance: participate in multi-layer review and automated checks.

Skills

5+ years software engineering

Native English fluency

CLI/workflow proficiency

Multilingual text processing

Education

Top-tier engineering university

Tools

Python

Shell scripting

Data processing

Job description

LILT is seeking experienced software engineers to design, build, and validate a multilingual benchmark suite for evaluating language models. You will create high-signal tasks in your native language, ensuring assets remain in the target language to measure multilingual handling without English translation crutches.

As a remote freelance contractor, you’ll perform task engineering, asset creation, prompting and translation, and robust verification scripts, while calibrating task difficulty across