LLM Data Scientist: End-to-End Training & Evaluation

Propio Language Services; Overland Park (KS); On-site; USD 120,000 - 180,000; Full time

Job summary

Propio Language Services seeks an Applied Scientist/Research Engineer, LLM Training Data to own multilingual data strategy and curation pipelines. You will shape annotation workflows, QA rubrics, and evaluation datasets to improve LLM performance, handling data from acquisition to post-training stages.

The role requires building scalable data pipelines, identifying model gaps, and driving data-driven improvements across multilingual AI systems within a fast-growing organization.

Qualifications

• Master’s degree in CS/ML/DS or equivalent practical experience.
• 4+ years in AI data, ML data operations, NLP data engineering, applied ML, or LLM data workflows.
• Strong Python and SQL skills; experience with dataset curation pipelines.
• Experience with annotation workflows, QA rubrics, evaluation datasets, or human-in-the-loop processes.
• Familiarity with multilingual NLP, speech/translation data, low-resource languages, or agentic AI datasets.
• Experience with AWS data and ML tools (S3, Glue, SageMaker, etc.).
• Excellent communication and cross-functional collaboration abilities.

Responsibilities

• Define the end-to-end data roadmap for multilingual and multimodal AI systems.
• Design and build dataset curation pipelines for training, post-training, and evaluation.
• Create annotation schemas, labeling guidelines, QA rubrics, and reviewer workflows.
• Build evaluation datasets and benchmarks; analyze model failure modes.
• Support post-training data workflows such as SFT, instruction tuning, and RLHF-style data.
• Utilize AWS-based data infrastructure to scale secure, traceable AI data workflows.

Skills

Communication skills

Cross-functional collaboration

Education

Master’s degree in CS/ML/DS

PhD in CS/ML/NLP (preferred)

Tools

S3

Glue

SageMaker

Bedrock

Lambda

Step Functions

EKS/ECS

IAM

KMS

Job description

Propio Language Services seeks an Applied Scientist/Research Engineer, LLM Training Data to own multilingual data strategy and curation pipelines. You will shape annotation workflows, QA rubrics, and evaluation datasets to improve LLM performance, handling data from acquisition to post-training stages.

The role requires building scalable data pipelines, identifying model gaps, and driving data-driven improvements across multilingual AI systems within a fast-growing organization.