Applied Scientist, Foundational AI & LLM Evaluation

Amazon; Boston (MA); On-site; USD 136,000 - 184,000; Full time

Job summary

Amazon.com Services LLC seeks an Applied Scientist for the AGI team to advance data quality in ML and multimodal systems, building auditing frameworks, SOPs, and evaluation pipelines for LLMs.

You will design judge architectures, run manual and meta-audits, coach testers, and partner with engineers and vendor teams to deliver high‑quality training data powering state‑of‑the‑art models. This role combines hands‑on research with scalable quality assurance across data collection workflows.

Qualifications

• Master's degree in CS, math, stats, ML or equivalent.
• Experience programming in Java, C++, Python.
• Experience with SQL and an RDBMS or Data Warehouse.

Responsibilities

• Develop quality strategies and auditing frameworks for data collection workflows.
• Design LLM evaluation architectures, judge architectures, and evaluation rubrics.
• Collaborate with engineers and vendor teams to implement automated quality systems.

Skills

Java

C++

Python

SQL

Education

Master's degree in computer science, mathematics, statistics, machine learning or equivalent quantitative field

Tools

Oracle

Job description

Amazon.com Services LLC seeks an Applied Scientist for the AGI team to advance data quality in ML and multimodal systems, building auditing frameworks, SOPs, and evaluation pipelines for LLMs.

You will design judge architectures, run manual and meta-audits, coach testers, and partner with engineers and vendor teams to deliver high‑quality training data powering state‑of‑the‑art models. This role combines hands‑on research with scalable quality assurance across data collection workflows.