AGI Applied Science Manager: Lead LLM Quality & Audits

Amazon; Bellevue (WA); On-site; USD 184,000 - 249,000; Full time

Benefits offered by this job

RSUs

Sign-on payments

401(k) matching

Health insurance

Job summary

Amazon is seeking an Applied Science Manager for the AGI frontier to lead a team of scientists focused on data quality, auditing, and evaluation for LLMs and multimodal systems. You will guide auditing strategies, develop evaluation rubrics, and build automated quality assessment models while recruiting and mentoring team members.

The role involves communicating with senior leadership, cross-functional teams, and customers to translate requirements into product designs, and to drive high-impact

Qualifications

• 3+ years of scientists or machine learning engineers management experience
• Knowledge of ML, NLP, Information Retrieval and Analytics

Responsibilities

• Lead and mentor a team of Applied Scientists who develop quality strategies and auditing frameworks for data collection workflows.
• Oversee manual audits, meta-audits, and coaching to uplift quality capabilities across the team.
• Design auditing strategies, SOPs, quality metrics, and sampling methodologies for AGI data quality and evaluation.
• Develop and maintain LLM-as-a-Judge systems, judge architectures, evaluation rubrics, and ML models for automated quality assessment.

Skills

ML knowledge

NLP

Information Retrieval

Analytics

Job description

Amazon is seeking an Applied Science Manager for the AGI frontier to lead a team of scientists focused on data quality, auditing, and evaluation for LLMs and multimodal systems. You will guide auditing strategies, develop evaluation rubrics, and build automated quality assessment models while recruiting and mentoring team members.

The role involves communicating with senior leadership, cross-functional teams, and customers to translate requirements into product designs, and to drive high-impact