Applied Scientist, AGI & LLM Quality & Auditing

Amazon; Bellevue, Northern (WA, KY); Hybrid; USD 136,000 - 184,000; Full time

Job summary

Amazon.com Services LLC is seeking an Applied Scientist for the Artificial General Intelligence (AGI) team to advance data quality in ML and LLM-based systems. You will design auditing strategies, develop automated evaluation tools, and build quality metrics and feedback loops across data collection workflows to strengthen Nova model performance.

Day-to-day work includes quality solution design, root-cause analysis, and collaboration with engineers, scientists, and vendors to deploy automated

Qualifications

• Master's degree in a quantitative field.
• Proficiency in Java, C++, and Python.
• Experience with SQL and an RDBMS (Oracle) or Data Warehouse.

Responsibilities

• Develop quality strategies and auditing frameworks for data collection workflows.
• Design and implement LLM-as-a-Judge systems and evaluation rubrics.
• Perform expert-level audits, meta-audits, and coaching to improve data quality.

Skills

Java

C++

Python

SQL

Education

Master's degree in computer science, mathematics, statistics, machine learning or equivalent quantitative field

Tools

Oracle

Data Warehouse

Job description

Amazon.com Services LLC is seeking an Applied Scientist for the Artificial General Intelligence (AGI) team to advance data quality in ML and LLM-based systems. You will design auditing strategies, develop automated evaluation tools, and build quality metrics and feedback loops across data collection workflows to strengthen Nova model performance.

Day-to-day work includes quality solution design, root-cause analysis, and collaboration with engineers, scientists, and vendors to deploy automated