AI Benchmark Engineer | Native Language Specialist - German (Austria) - Remote

lilt-production; United States; Remote; USD 78,000 - 134,000; Part time

Benefits offered by this job

Flexible hours

Prompt payment

Fully remote freelance engagement

Access to a global linguist community

Job summary

lilt-production is seeking a remote, freelance Engineer to design and validate evaluation benchmarks for multilingual software tasks in German (Austria). You will craft tests that surface German-language and encoding edge cases in terminal workflows, focusing on robustness rather than translation.

You will build realistic task environments, write verifier scripts, and run benchmarks across model tiers, contributing to quality-control rounds with human and automated checks.

Qualifications

• Five or more years of industry software engineering experience.
• Native or near-native German (Austria) fluency with high English proficiency.
• Strong proficiency in Python and shell scripting.
• Extensive experience with multilingual text-processing pitfalls, including Unicode normalization, locale-dependent collation and casing.
• Experience with Terminal/CLI-based development workflows.

Responsibilities

• Engineer Terminal-Bench tasks that probe how coding agents handle German-language inputs, locale conventions, and encoding subtleties.
• Build realistic task environments whose datasets, files, and artifacts remain in the target language to surface real multilingual failure modes.
• Identify prompting and translation failure points where current models degrade in German, then craft tasks that expose them.
• Develop reference implementations and write deterministic verifier scripts, using rubric-based judging only when strictly necessary.
• Run tasks against multiple model tiers, analyze execution logs, and calibrate difficulty from Easy through Very Hard.
• Contribute to a multi-layer quality control process combining human review, calibration review, audit, and automated model-based checks.

Skills

Python

Shell scripting

Data processing

Terminal/CLI workflows

Multilingual text-processing basics

Education

Top-tier engineering university

Tools

CLI tools

Job description

Role overview

A remote, freelance position focused on designing and validating evaluation benchmarks that stress-test large language models on multilingual software tasks in German (Austria). The work targets prompt-language effects, non-English data handling, and locale/encoding edge cases inside terminal workflows, with the goal of producing high-signal tasks that genuinely measure multilingual robustness rather than English-translation workarounds.

Responsibilities

• Engineer Terminal-Bench tasks that probe how coding agents handle German-language inputs, locale conventions, and encoding subtleties.
• Build realistic task environments whose datasets, files, and artifacts remain in the target language to surface real multilingual failure modes.
• Identify prompting and translation failure points where current models degrade in German, then craft tasks that expose them.
• Develop reference implementations and write deterministic verifier scripts, using rubric-based judging only when strictly necessary.
• Run tasks against multiple model tiers, analyze execution logs, and calibrate difficulty from Easy through Very Hard.
• Contribute to a multi-layer quality control process combining human review, calibration review, audit, and automated model-based checks.

Requirements

• Five or more years of industry software engineering experience.
• Background at leading technology companies and/or graduation from a top-tier engineering university.
• Native or near-native German (Austria) fluency with deep understanding of grammar, register, and phrasing, plus high English proficiency.
• Strong proficiency in Python, standard shell scripting, and data processing.
• Extensive experience with Terminal/CLI-based development workflows and working familiarity with coding agents.
• Deep technical grasp of multilingual text-processing pitfalls, including Unicode normalization, locale-dependent collation and casing, non-Gregorian date formats, safe string operations, and toolchain I/O interoperability.

Nice to have

• Familiarity with bidirectional/RTL handling, font fallbacks, and typography considerations for UI or generated artifacts.

Benefits and work setup

• Fully remote independent-contractor engagement with flexible hours and no fixed schedule.
• Competitive contractor rates with prompt payment and no invoice chasing.
• Best treated as supplemental income rather than a primary role, since project availability fluctuates.
• Access to a global community of linguists and subject‑matter experts working on applied AI and language technology.
• Geographic restrictions may apply based on international sanctions or embargoes; contractors handle their own tax obligations.