Frontier AI Data Quality Scientist: Auto & Self-Improving

Amazon Inc.; Bellevue (WA); On-site; USD 167,000 - 226,000; Full time

Benefits offered by this job

Health insurance

401(k) matching

Paid time off

Job summary

Amazon.com Services LLC is seeking a Senior Applied Scientist for the Frontier AI Assets – Assessments team in Bellevue, WA. You will design automated systems to assess data quality for frontier models and develop agents that audit themselves, aiming to improve accuracy over time.

You will lead methodology design, calibration against expert judgment, and publish methods while mentoring junior scientists and communicating outcomes to product and leadership teams.

Qualifications

• 5+ years building ML models for business applications.
• Experience programming in Java, C++, Python or related language.
• Experience with neural deep learning methods and machine learning.
• PhD in Computer Science, Engineering, Mathematics, Statistics, Economics, or a related field.
• Experience with ML and Large Language Model fundamentals, including architecture, training/inference lifecycles, and optimization of model execution, or experience building complex software systems that have been successfully delivered to customers.
• Experience in ML, data mining, information retrieval, statistics or natural language processing, or experience designing or architecting (design patterns, reliability and scaling) of new and existing systems.

Responsibilities

• Design the assessment methodology for frontier AI assets.
• Build the automated assessment system, from LLM-as-a-Judge baselines to agents that plan their own audits and learn from their own errors.
• Oversee the expert audit program and raise the proficiency of the auditors who run it.
• Design the measurement methods that make quality findings defensible, including sampling strategy and error analysis.
• Communicate findings to the teams that create the data and to the teams that train models and build agents using it.
• Set technical direction for assessment science, mentor junior scientists, and present results to senior leadership.
• Publish research on assessment methodology at top-tier venues and contribute to patents.

Skills

ML model development

Java

C++

Python

Deep learning

Education

PhD in Computer Science or related field

Tools

TensorFlow

PyTorch

Job description

Amazon.com Services LLC is seeking a Senior Applied Scientist for the Frontier AI Assets – Assessments team in Bellevue, WA. You will design automated systems to assess data quality for frontier models and develop agents that audit themselves, aiming to improve accuracy over time.

You will lead methodology design, calibration against expert judgment, and publish methods while mentoring junior scientists and communicating outcomes to product and leadership teams.