Applied Scientist II: LLM Shopping Metrics

Amazon; San Francisco (CA); On-site; USD 143,000 - 193,000; Full time

Benefits offered by this job

RSUs

Health insurance

401(k) matching

Job summary

Amazon is seeking an Applied Scientist II to own LLM-based labeling for perception-driven defects within Core Shopping Data Science. You'll design prompts, sampling strategies, and balance model judgment with human annotation, validating against ground truth and maintaining gold datasets.

You will extend the measurement across shopping experiences, build reusable labeling and evaluation tooling, and define standards for teams using LLMs to measure customer experience.

Qualifications

• PhD, or Master's degree and 4+ years of CS, CE, ML or related field experience.
• Experience programming in Java, C++, Python or related language.
• Experience in algorithms, data structures, parsing, numerical optimization, data mining, parallel and distributed computing, high-performance computing.

Responsibilities

• Design and improve LLM-based labeling for perception-driven defects: prompt design, sampling strategy, and model vs human annotation.
• Validate labeling quality against human ground truth and maintain golden datasets.
• Extend measurement to other parts of the shopping experience and design methodologies where none exists.
• Build reusable, production-grade labeling and evaluation tooling, batch inference, quality sampling, prompt and model version control.

Skills

Java

C++

Python

Unix/Linux

Education

PhD or MS + 4+ years CS/CE/ML

Tools

Unix/Linux

Job description

Amazon is seeking an Applied Scientist II to own LLM-based labeling for perception-driven defects within Core Shopping Data Science. You'll design prompts, sampling strategies, and balance model judgment with human annotation, validating against ground truth and maintaining gold datasets.

You will extend the measurement across shopping experiences, build reusable labeling and evaluation tooling, and define standards for teams using LLMs to measure customer experience.