AGI Data Quality & Auditing Manager

Amazon Inc.; Omaha, Northern (NE, KY); Hybrid; USD 184,000 - 249,000; Full time

Job summary

Amazon.com Services LLC is seeking an Applied Science Manager for the AGI team (Frontier AI Assets: Assessments) to lead a group of scientists shaping auditing frameworks and quality strategies for data used in training and evaluating LLMs and multimodal systems.

The role focuses on designing judge architectures, evaluation rubrics, and automated QA, while guiding recruitment, performance reviews, and cross-functional collaboration to deliver high-quality training data and robust AI products.

Qualifications

• 3+ years of scientists or machine learning engineers management experience.
• Knowledge of ML, NLP, Information Retrieval and Analytics.

Responsibilities

• Lead and mentor a team of Applied Scientists to design comprehensive quality strategies and auditing frameworks.
• Oversee development of LLM-as-a-Judge systems, evaluation rubrics, and automated quality assessment models.
• Coordinate data collection workflows and communicate quality feedback to stakeholders.
• Recruit, hire, and develop team members, conduct performance reviews, and align with organizational goals.

Skills

Management experience

ML, NLP, IR & Analytics

Job description

Amazon.com Services LLC is seeking an Applied Science Manager for the AGI team (Frontier AI Assets: Assessments) to lead a group of scientists shaping auditing frameworks and quality strategies for data used in training and evaluating LLMs and multimodal systems.

The role focuses on designing judge architectures, evaluation rubrics, and automated QA, while guiding recruitment, performance reviews, and cross-functional collaboration to deliver high-quality training data and robust AI products.