AI Benchmark Engineer | Native Language Specialist - Indonesian - Remote

LILT (Production); Indonesia; Remote; IDR 620.429.000 - 1.489.029.000; Vollzeit

Zusammenfassung

LILT (Production) seeks experienced software engineers to design and validate multilingual benchmarks. You will create high-signal tasks in your native language and test model robustness across non-English data and locale encodings. This is a remote, freelance opportunity designed for experts in multilingual software challenges.

The role emphasizes asset creation, evaluation, and deterministic verifier scripts, with a rigorous 4-layer quality control process to ensure benchmark integrity.

Qualifikationen

• Experience: 1+ years in software engineering.
• Background: Proven track record at leading technology companies or graduation from top-tier engineering universities.
• Language: Native or near-native fluency, with a deep understanding of its grammar.

Aufgaben

• Task Engineering: Evaluating Coding Agents.
• Asset Creation: Build realistic task environments using datasets and files in your native language, assets remain in the target language.
• Prompting & Translation: identify failure points where AI does not work, in your native language.
• Implementation & Verification: support robust solutions and write deterministic verifier scripts when needed.
• Calibration & Execution: analyze execution logs and calibrate task difficulty across model tiers.
• Quality Assurance: participate in four-layer quality control and automated checks.

Kenntnisse

Software engineering

Ausbildung

Top-tier engineering university

Jobbeschreibung

About The Opportunity

We are building a rigorous, verifiable evaluation suite of Terminal-Bench tasks designed to test the limits of large language models on multilingual software challenges. Our goal is to measure multilingual robustness across prompt language effects, non-English data processing, and complex locale/encoding edge cases in terminal workflows. We are seeking experienced native-speaking software engineers to design, build, and validate these benchmarks. You will create high-signal, high-quality tasks that genuinely test a model's ability to handle multilingual environments without relying on English translation crutches. Note this is a remote, freelance opportunity

What You’ll Deliver

• Task Engineering: Evaluating Coding Agents.
• Asset Creation: Build realistic task environments using datasets and files in your native language. Crucially, these assets must remain in the target language to genuinely measure multilingual handling.
• Prompting & Translation: finding failure points where AI does not work, in your native language Implementation & Verification: Support the development of robust solutions (reference implementations) and write highly reliable, deterministic verifier scripts (using rubric-based judging only when strictly necessary).
• Calibration & Execution: Analyze execution logs and calibrate task difficulty (Easy to Very Hard) using standard Terminal-Bench run configurations against various model tiers (Haiku, Sonnet, Opus).
• Quality Assurance: Participate in a rigorous, 4-layer human quality control process (creation, human review, calibration review, and audit) alongside automated LLM-based checks to ensure fairness, grammatical accuracy, and benchmark integrity.

Qualifications

• Experience: 1+ years of industry experience in software engineering.
• Background: Proven track record at leading technology companies and/or graduation from top-tier engineering universities.
• Language: Native or near-native fluency, with a deep understanding of its gramma