Remote AI Benchmark Engineer: German (Austria) Expert

lilt-production; United States; Remote; USD 78,000 - 134,000; Part time

Benefits offered by this job

Flexible hours

Prompt payment

Fully remote freelance engagement

Access to a global linguist community

Job summary

lilt-production is seeking a remote, freelance Engineer to design and validate evaluation benchmarks for multilingual software tasks in German (Austria). You will craft tests that surface German-language and encoding edge cases in terminal workflows, focusing on robustness rather than translation.

You will build realistic task environments, write verifier scripts, and run benchmarks across model tiers, contributing to quality-control rounds with human and automated checks.

Qualifications

• Five or more years of industry software engineering experience.
• Native or near-native German (Austria) fluency with high English proficiency.
• Strong proficiency in Python and shell scripting.
• Extensive experience with multilingual text-processing pitfalls, including Unicode normalization, locale-dependent collation and casing.
• Experience with Terminal/CLI-based development workflows.

Responsibilities

• Engineer Terminal-Bench tasks that probe how coding agents handle German-language inputs, locale conventions, and encoding subtleties.
• Build realistic task environments whose datasets, files, and artifacts remain in the target language to surface real multilingual failure modes.
• Identify prompting and translation failure points where current models degrade in German, then craft tasks that expose them.
• Develop reference implementations and write deterministic verifier scripts, using rubric-based judging only when strictly necessary.
• Run tasks against multiple model tiers, analyze execution logs, and calibrate difficulty from Easy through Very Hard.
• Contribute to a multi-layer quality control process combining human review, calibration review, audit, and automated model-based checks.

Skills

Python

Shell scripting

Data processing

Terminal/CLI workflows

Multilingual text-processing basics

Education

Top-tier engineering university

Tools

CLI tools

Job description

lilt-production is seeking a remote, freelance Engineer to design and validate evaluation benchmarks for multilingual software tasks in German (Austria). You will craft tests that surface German-language and encoding edge cases in terminal workflows, focusing on robustness rather than translation.

You will build realistic task environments, write verifier scripts, and run benchmarks across model tiers, contributing to quality-control rounds with human and automated checks.