Senior Scientist, Frontier AI Data-Quality Assessments

Amazon Inc.; Seattle, Northern (WA, KY); Hybrid; USD 167,000 - 226,000; Full time

Benefits offered by this job

Health insurance

401(k) matching

Paid time off

Parental leave

Job summary

Amazon.com Services LLC's Frontier AI Assets (FAI) Assessment team seeks a Senior Applied Scientist to advance automated data quality measurement for frontier models. You will design methodologies, build systems from LLM-as-a-Judge baselines to self-planning auditors, and mentor junior scientists.

This role emphasizes scalable evaluation, rigorous experiments, and publication of results in top venues. Collaboration with data, modeling, and engineering teams is expected.

Qualifications

• PhD in Computer Science, Engineering, Mathematics, Statistics, Economics, or a related field.
• 5+ years building ML models for business or research use.
• Experience with ML, deep learning, LLMs, and system design for production.

Responsibilities

• Design assessment methodologies for frontier AI assets and datasets.
• Build automated assessment systems from LLM baselines to self-auditing agents.
• Oversee expert audit programs and uplift auditors' proficiency.
• Develop measurement methods with defensible findings and error analysis.
• Communicate findings to data creators and model/training teams.
• Set technical direction for assessment science and mentor junior scientists.
• Publish research on assessment methodology at top-tier venues.

Skills

Java

C++

Python

Deep learning

ML fundamentals

NLP

Java/C++/Python

Education

PhD in Computer Science, Engineering, Mathematics, Statistics, Economics, or a related field

Tools

TensorFlow

PyTorch

Job description

Amazon.com Services LLC's Frontier AI Assets (FAI) Assessment team seeks a Senior Applied Scientist to advance automated data quality measurement for frontier models. You will design methodologies, build systems from LLM-as-a-Judge baselines to self-planning auditors, and mentor junior scientists.

This role emphasizes scalable evaluation, rigorous experiments, and publication of results in top venues. Collaboration with data, modeling, and engineering teams is expected.