Remote AI Benchmark Engineer - Spanish Native

LILT (Production); Argentina; Remote; ARS 41,729,000 - 83,458,000; Full time

Job summary

LILT Production in Argentina is seeking experienced software engineers to design, build, and validate multilingual benchmarks. You will create high-signal tasks in native languages, test models on non-English data, and ensure the assets stay in the target language.

This remote, freelance role emphasizes rigorous QA, deterministic verifiers, and calibration across model tiers, with emphasis on grammar accuracy and locale handling.

Qualifications

• Experience: 1+ years of industry experience in software engineering.
• Background: Proven track record at leading technology companies and/or graduation from top-tier engineering universities.
• Language: Native or near-native fluency, with a deep understanding of its gramma

Responsibilities

• Task Engineering: Evaluating Coding Agents.
• Asset Creation: Build realistic task environments using datasets and files in your native language. Crucially, these assets must remain in the target language to genuinely measure multilingual handling.
• Prompting & Translation: finding failure points where AI does not work, in your native language Implementation & Verification: Support the development of robust solutions (reference implementations) and write highly reliable, deterministic verifier scripts (using rubric-based judging only when strictly necessary).
• Calibration & Execution: Analyze execution logs and calibrate task difficulty (Easy to Very Hard) using standard Terminal-Bench run configurations against various model tiers (Haiku, Sonnet, Opus).
• Quality Assurance: Participate in a rigorous, 4-layer human quality control process (creation, human review, calibration review, and audit) alongside automated LLM-based checks to ensure fairness, grammatical accuracy, and benchmark integrity.

Skills

Software engineering

Native fluency

Job description

LILT Production in Argentina is seeking experienced software engineers to design, build, and validate multilingual benchmarks. You will create high-signal tasks in native languages, test models on non-English data, and ensure the assets stay in the target language.

This remote, freelance role emphasizes rigorous QA, deterministic verifiers, and calibration across model tiers, with emphasis on grammar accuracy and locale handling.