Applied Scientist, Foundational AI & LLMs

Amazon Science; Bellevue (WA); On-site; USD 136,000 - 184,000; Full time

Benefits offered by this job

Health insurance

RSUs

401(k) matching

Job summary

Amazon Science is seeking an Applied Scientist for the AGI team to lead quality strategies, auditing frameworks, and automated evaluation systems for data used in LLMs and multimodal models.

You will design judge architectures, evaluation rubrics, and build ML models for automated quality assessment, while coaching teams to uplift overall quality capabilities and safeguard data integrity.

Qualifications

• Master's degree in computer science, mathematics, statistics, machine learning or equivalent quantitative field.
• Experience programming in Java, C++, Python or related language.
• Experience with SQL and an RDBMS (e.g., Oracle) or Data Warehouse.

Responsibilities

• Develop quality strategies and auditing frameworks for data collection workflows.
• Design auditing meta-audits and coaching to improve auditor performance.
• Build LLM-as-a-Judge systems with evaluation rubrics and ML models for automated quality assessment.
• Configure data collection workflows and communicate quality feedback to stakeholders.
• Enhance customer experiences through high-quality training data powering state-of-the-art LLM products.

Skills

Java

C++

Python

SQL

Education

Master's degree in CS/Math/Statistics/ML

Tools

Oracle

Data Warehouse

Job description

Amazon Science is seeking an Applied Scientist for the AGI team to lead quality strategies, auditing frameworks, and automated evaluation systems for data used in LLMs and multimodal models.

You will design judge architectures, evaluation rubrics, and build ML models for automated quality assessment, while coaching teams to uplift overall quality capabilities and safeguard data integrity.