AI Research Engineer

RWS; Polska; On-site; PLN 180,000 - 300,000; Full time

Job summary

RWS is seeking an AI Research Engineer to bridge applied research and engineering in the TrainAI team. You will design, build, and scale continuous LLM evaluation pipelines, experiment orchestration, and automated pre-annotation workflows.

You will work with the Head of Innovation and data engineering to turn cutting-edge multilingual evaluation methods into reliable tools for research and production. The role focuses on automating evaluation, benchmarking across languages, and translating

Qualifications

• Master's degree in a technical field and 3–5+ years of NLP/ML engineering experience.
• Strong programming in Python and DL/NLP frameworks.
• Experience with LLM orchestration and automated evaluation techniques.
• Experience with multilingual NLP, data processing, and cloud ML workflows.

Responsibilities

• Design, build, and maintain automated execution pipelines for continuous LLM benchmarking across languages, tasks, and models.
• Develop and roll out LLM-assisted pre-annotation and QA automation pipelines to automate production data annotation.
• Implement and refine automated evaluation methodologies, including LLM-as-a-judge pipelines, metrics, and scoring algorithms.
• Build experiment infrastructure, datasets, and execution pipelines to support multilingual LLM research studies.
• Translate applied NLP research into scalable prototypes and integrate research tools into production operations.
• Collaborate on writing up research findings and benchmark reports for internal and public release.
• Prototype novel capabilities in agentic AI evaluation and synthetic data quality certification.

Skills

Python

PyTorch

TensorFlow

Hugging Face transformers

LangChain

LlamaIndex

Multilingual NLP

NLP/ML engineering

Education

Master’s degree in Computer Science / AI / Computational Linguistics

Tools

LangChain

LlamaIndex

Job description

As an AI Research Engineer in the TrainAI research team, you will bridge applied research and engineering execution. Your core focus will be designing, building, and scaling continuous LLM evaluation pipelines, experiment orchestration tools, and automated pre-annotation workflows. Working directly with the Head of Innovation and data engineering teams, you will turn cutting‑edge multilingual evaluation methodologies into reliable, automated research and production tools.

Key Responsibilities

• Evaluation Pipelines & Benchmarking: Design, build, and maintain automated execution pipelines for continuous LLM benchmarking (such as TrainAI Bench) across diverse languages, tasks, and models.
• Pre-Annotation & QA Automation: Develop and roll out LLM‑assisted pre‑annotation and quality assurance pipelines to automate production data annotation, lowering costs and improving data consistency.
• LLM‑as‑a‑Judge Frameworks: Implement and refine automated evaluation methodologies, including LLM‑as‑a‑judge pipelines, custom evaluation metrics, and scoring algorithms.
• Experiment Orchestration & Research Support: Build experiment infrastructure, datasets, and execution pipelines to support flagship multilingual LLM research studies and thought leadership reports.
• Research‑to‑Engineering Bridge: Translate applied NLP research into scalable prototypes and work alongside the core data engineering team to integrate research tools into production operations.
• Thought Leadership & Publication: Collaborate with the research team on writing up research findings, benchmark reports, and thought leadership pieces for internal and public release.
• Emerging AI Methods Exploration: Prototype novel capabilities in emerging areas such as agentic AI evaluation, synthetic data quality certification, and domain‑specific LLM assessment.
• Documentation & Best Practices: Maintain comprehensive documentation, code bases, and technical guides for TrainAI evaluation frameworks and tools.

Skills & Experience

• Education & Experience: Master’s degree (MSc or equivalent) in Computer Science, Artificial Intelligence, Computational Linguistics, or a related technical field, combined with 3–5+ years of hands‑on experience in NLP or ML engineering.
• Programming & ML Stack: Advanced proficiency in Python and deep learning/NLP frameworks (PyTorch or TensorFlow, Hugging Face transformers).
• LLM Tooling & Frameworks: Proven experience with LLM orchestration frameworks (e.g., LangChain, LlamaIndex) and automated evaluation techniques (e.g., LLM‑as‑a‑judge, benchmarking harnesses).
• Data & Analytics Libraries: Expertise in data manipulation, structured evaluation data processing, and analysis using pandas, NumPy, or related tools.
• Cloud Infrastructure: Practical experience deploying and managing ML workflows on cloud platforms (Azure, AWS, or Vertex AI).
• System Design: Experience building scalable data or experiment orchestration pipelines.
• Familiarity with multilingual NLP, machine translation, or computational linguistics (e.g., cross‑linguistic variations, syntax, morphology).
• Knowledge of localization workflows, annotation platforms, or human‑in‑the‑loop data pipelines.
• Track record of contributing to open‑source NLP projects, evaluation frameworks, or published technical benchmarks.

Life at RWS

If you like the idea of working with smart people who are passionate about growing the value of ideas, data and content by making sure organizations are understood , then you’ll love life at RWS.

Our purpose is to unlock global understanding. This means our work fundamentally recognizes the value of every language and culture. So, we celebrate difference, we are inclusive and believe that diversity makes us strong. We want every employee to grow as an individual and excel in their career.

In return, we expect all our people to live by the values that unite us: to partner, putting clients fist and winning together, to pioneer, innovating fearlessly and leading with vision and courage, to progress, aiming high and growing through actions and to deliver, owning the outcome and building trust with our colleagues and clients.

RWS Values

Get the 3Ps right – Partner, Pioneer, Progress – and we´ll Deliver together as RWS.

RWS embraces DEI and promotes equal opportunity, we are an Equal Opportunity Employer and prohibit discrimination and harassment of any kind. RWS is committed to the principle of equal employment opportunity for all employees and to providing employees with a work environment free of discrimination and harassment. All employment decisions at RWS are based on business needs, job requirements and individual qualifications, without regard to race, religion, nationality, ethnicity, sex, age, disability, or sexual orientation. RWS will not tolerate discrimination based on any of these characteristics.