Multilingual LLM Data Engineer & Evaluation Scientist

Propio; Overland Park (KS); Hybrid; USD 120,000 - 180,000; Full time

Job summary

Propio Language Services is a top-five global language services provider delivering multilingual interpretation, translation, and localization at scale. We are hiring an Applied Scientist/Research Engineer, LLM Training Data to own data strategy, curation pipelines, annotation workflows, and evaluation datasets powering our multilingual AI systems.

This hands-on role requires building scalable data pipelines, designing QA processes, identifying model failure modes, and improving LLM performance

Qualifications

• Master’s degree in CS, ML, or related field, or equivalent practical experience.
• 4+ years of experience in AI data operations, NLP data engineering, or LLM data workflows.
• Strong hands-on experience with Python, SQL, and dataset curation pipelines.
• Experience with annotation workflows, QA rubrics, evaluation datasets, or human-in-the-loop data processes.
• Familiarity with multilingual NLP, speech data, translation data, low-resource languages, or agentic AI datasets.
• Working knowledge of AWS data/ML tools (SageMaker, Glue, S3, Bedrock, Lambda, IAM, KMS).

Responsibilities

• Define the end-to-end data roadmap for multilingual and multimodal AI systems.
• Design and build dataset curation pipelines for training, post-training, and evaluation.
• Create annotation schemas, labeling guidelines, QA rubrics, golden datasets, and reviewer workflows.
• Build evaluation datasets and benchmarks, analyze model failure modes, and translate gaps into data improvements.
• Support post-training data workflows such as SFT, instruction tuning, RLHF, and synthetic data generation.
• Use modern annotation tools and AWS-based data infrastructure to scale secure AI data workflows.

Skills

Python

SQL

Data pipelines

Communication skills

NLP data

Education

Master's degree in CS/ML or related field

PhD in CS/ML or related field

Tools

SageMaker

Bedrock

S3

Glue

Lambda

IAM

KMS

EKS/ECS

Labelbox

Job description

Propio Language Services is a top-five global language services provider delivering multilingual interpretation, translation, and localization at scale. We are hiring an Applied Scientist/Research Engineer, LLM Training Data to own data strategy, curation pipelines, annotation workflows, and evaluation datasets powering our multilingual AI systems.

This hands-on role requires building scalable data pipelines, designing QA processes, identifying model failure modes, and improving LLM performance