LLM Scientist — Conversational AI & Evaluation

Amazon; Greater London; On-site; GBP 90,000 - 130,000; Full time

Job summary

Amazon is seeking a Data Scientist to help build industry-leading language technology powering Rufus, our AI-driven search and shopping assistant. The role focuses on developing and optimizing LLM-powered conversational experiences, with an emphasis on instruction design, contextual grounding, and model evaluation.

You will work with vast data to improve conversational AI performance at scale, collaborating with scientists and engineers to push the boundaries of NLP, ML, and information

Qualifications

• Strong data science background with ML/analytics focus.
• Experience with natural language processing and large language model concepts.
• Ability to design experiments, evaluate model performance, and apply results to product goals.

Responsibilities

• Develop and maintain LLM instructions iterations and evaluation frameworks.
• Partner with the wider org to experiment with retrieval augmentation and prompt decomposition.
• Design and analyze A/B tests to evaluate new features and model improvements.
• Develop metrics, dashboards, and reporting for system performance and business impact.
• Collaborate with Applied Scientists and Engineers to translate insights into production systems.
• Establish automated processes for large‑scale data analysis, ETL pipelines, and experimentation frameworks.
• Communicate results to technical and non‑technical audiences through presentations and reports.

Skills

Data science

Machine learning

NLP

Job description

Amazon is seeking a Data Scientist to help build industry-leading language technology powering Rufus, our AI-driven search and shopping assistant. The role focuses on developing and optimizing LLM-powered conversational experiences, with an emphasis on instruction design, contextual grounding, and model evaluation.

You will work with vast data to improve conversational AI performance at scale, collaborating with scientists and engineers to push the boundaries of NLP, ML, and information