Remote Native-Language AI Benchmark Engineer

Openkrill; United States; Remote; USD 83,000 - 165,000; Full time

Job summary

Openkrill is seeking experienced software engineers to design, build, and validate a multilingual benchmark suite for testing LLMs in terminal workflows. You will create high-signal, native-language tasks that avoid English translation crutches and push model robustness.

You’ll implement reference solutions, write deterministic verifier scripts, and calibrate difficulty across model tiers while participating in a rigorous, four-layer quality-control process to ensure accuracy and fairness.

Qualifications

• 5+ years of software engineering experience.
• Proven track record at leading technology companies or top-tier universities.
• Native or near-native fluency in target language with high English proficiency.
• Strong Python, shell scripting, and data processing skills.
• Experience with Terminal/CLI workflows and coding agents.
• Deep understanding of multilingual text processing, including Unicode normalization and locale conventions.

Responsibilities

• Design and build multilingual benchmarks.
• Create high-signal tasks in native language.
• Identify failure points in native language.
• Develop reference implementations and deterministic verifier scripts.
• Calibrate task difficulty across model tiers.
• Participate in a four-layer QA process and automated checks.

Skills

5+ years software engineering

Native or near-native fluency

Python

Shell scripting

Data processing

Terminal/CLI workflows

Coding agents

Multilingual text processing

Tools

Unix

Regex

Job description

Openkrill is seeking experienced software engineers to design, build, and validate a multilingual benchmark suite for testing LLMs in terminal workflows. You will create high-signal, native-language tasks that avoid English translation crutches and push model robustness.

You’ll implement reference solutions, write deterministic verifier scripts, and calibrate difficulty across model tiers while participating in a rigorous, four-layer quality-control process to ensure accuracy and fairness.