Applied Scientist, AGI: LLM Quality & Evaluation

Amazon; Boston (MA); On-site; USD 136,000 - 184,000; Full time

Benefits offered by this job

Health insurance

RSU eligible

Sign-on bonus

Job summary

Amazon’s Artificial General Intelligence (AGI) team seeks an Applied Scientist with a robust ML background to build data-quality systems for LLMs and multimodal models. This role focuses on auditing methodologies, automated evaluation, and high-quality training data powering next-gen Amazon Nova models.

You will design quality workflows, perform audits, mentor engineers, and collaborate with vendor teams to translate quality into scalable practice across projects.

Qualifications

• Master's degree in computer science, mathematics, statistics, ML or equivalent quantitative field.
• Experience programming in Java, C++, Python or related language.
• Experience with SQL and an RDBMS (e.g., Oracle) or Data Warehouse.

Responsibilities

• Develop quality strategies and auditing frameworks to safeguard data collection workflows.
• Design auditing strategies with SOPs, quality metrics, and sampling methodologies for benchmarking.
• Lead manual audits, meta-audits, and coaching to uplift quality capabilities.
• Develop and maintain LLM-as-a-Judge systems, judge architectures, evaluation rubrics, and ML models for quality assessment.
• Configure data collection workflows and communicate quality feedback to stakeholders.
• Impact customer experiences through high-quality training and evaluation data for LLM products.

Skills

Java

C++

Python

SQL

Education

Master's degree in CS or related

Tools

Oracle

Data Warehouse

Job description

Amazon’s Artificial General Intelligence (AGI) team seeks an Applied Scientist with a robust ML background to build data-quality systems for LLMs and multimodal models. This role focuses on auditing methodologies, automated evaluation, and high-quality training data powering next-gen Amazon Nova models.

You will design quality workflows, perform audits, mentor engineers, and collaborate with vendor teams to translate quality into scalable practice across projects.