Korean Language Expert for AI Data & Evaluation

Agency; United States; Remote; USD 90,000 - 120,000; Full time

Job summary

Agency is seeking a linguistics-focused evaluator to interact with language models, assess their outputs, and document failure modes. This role emphasizes thorough analysis of Korean grammar and pragmatics, with a focus on transparent reasoning and show-your-work demonstrations.

The candidate will challenge models on dialects and form, propose improvements to prompts and evaluation metrics, and contribute to higher-quality training data for AI systems.

Qualifications

• Expertise in Korean grammar, syntax, morphology, phonetics, semantics, and pragmatics.
• Clear, metacognitive communication and "show your work".
• Documentation of linguistic errors and reasoning failures.

Responsibilities

• Converse with language models on various language scenarios.
• Verify factual accuracy and logical soundness of model outputs.
• Capture reproducible error traces and document failure modes.
• Suggest improvements to prompt engineering and evaluation metrics.
• Challenge models on honorifics, syntax, morphology, and regional dialects.

Skills

Korean grammar

Linguistic pragmatics

Metacognitive communication

Job description

Agency is seeking a linguistics-focused evaluator to interact with language models, assess their outputs, and document failure modes. This role emphasizes thorough analysis of Korean grammar and pragmatics, with a focus on transparent reasoning and show-your-work demonstrations.

The candidate will challenge models on dialects and form, propose improvements to prompts and evaluation metrics, and contribute to higher-quality training data for AI systems.