Multilingual AI Evaluation Architect

RWS; Polska; On-site; PLN 180,000 - 300,000; Full time

Job summary

RWS is seeking an AI Research Engineer to bridge applied research and engineering in the TrainAI team. You will design, build, and scale continuous LLM evaluation pipelines, experiment orchestration, and automated pre-annotation workflows.

You will work with the Head of Innovation and data engineering to turn cutting-edge multilingual evaluation methods into reliable tools for research and production. The role focuses on automating evaluation, benchmarking across languages, and translating

Qualifications

• Master's degree in a technical field and 3–5+ years of NLP/ML engineering experience.
• Strong programming in Python and DL/NLP frameworks.
• Experience with LLM orchestration and automated evaluation techniques.
• Experience with multilingual NLP, data processing, and cloud ML workflows.

Responsibilities

• Design, build, and maintain automated execution pipelines for continuous LLM benchmarking across languages, tasks, and models.
• Develop and roll out LLM-assisted pre-annotation and QA automation pipelines to automate production data annotation.
• Implement and refine automated evaluation methodologies, including LLM-as-a-judge pipelines, metrics, and scoring algorithms.
• Build experiment infrastructure, datasets, and execution pipelines to support multilingual LLM research studies.
• Translate applied NLP research into scalable prototypes and integrate research tools into production operations.
• Collaborate on writing up research findings and benchmark reports for internal and public release.
• Prototype novel capabilities in agentic AI evaluation and synthetic data quality certification.

Skills

Python

PyTorch

TensorFlow

Hugging Face transformers

LangChain

LlamaIndex

Multilingual NLP

NLP/ML engineering

Education

Master’s degree in Computer Science / AI / Computational Linguistics

Tools

LangChain

LlamaIndex

Job description

RWS is seeking an AI Research Engineer to bridge applied research and engineering in the TrainAI team. You will design, build, and scale continuous LLM evaluation pipelines, experiment orchestration, and automated pre-annotation workflows.

You will work with the Head of Innovation and data engineering to turn cutting-edge multilingual evaluation methods into reliable tools for research and production. The role focuses on automating evaluation, benchmarking across languages, and translating