Applied Scientist — AGI, LLM Quality & Evaluation

Amazon; Bellevue, Northern (WA, KY); Hybrid; USD 136,000 - 184,000; Full time

Benefits offered by this job

RSUs

Sign-on bonus

Health insurance

401(k) matching

Parental leave

Job summary

Amazon AGI team seeks an Applied Scientist to advance data quality for Nova models and multimodal LLM systems. You will build comprehensive quality strategies, auditing frameworks, and automated evaluation tools to ensure top-tier training data and model outputs.

You will mentor engineers and vendor teams, design judge architectures, and drive improvements in data collection workflows and feedback loops for superior AI products.

Qualifications

• Master's degree in CS, math, stats, ML or equivalent.
• Experience programming in Java, C++, Python or related language.
• Experience with SQL and an RDBMS (e.g., Oracle) or Data Warehouse.
• Experience implementing algorithms using toolkits and self-developed code.

Responsibilities

• Lead the development of quality strategies and auditing frameworks.
• Design auditing strategies with SOPs, quality metrics, and sampling methods.
• Perform expert-level manual audits and meta-audits; coach auditors.
• Develop and maintain LLM‑as‑a‑Judge systems and evaluation rubrics.

Skills

Auditing methodologies

Quality assurance

Meta-audits

Data quality

Education

Master's degree in computer science, mathematics, statistics, machine learning or equivalent quantitative field

Tools

Java

C++

Python

SQL

Oracle

Job description

Amazon AGI team seeks an Applied Scientist to advance data quality for Nova models and multimodal LLM systems. You will build comprehensive quality strategies, auditing frameworks, and automated evaluation tools to ensure top-tier training data and model outputs.

You will mentor engineers and vendor teams, design judge architectures, and drive improvements in data collection workflows and feedback loops for superior AI products.