Applied Scientist, AGI & LLM Evaluation & Quality

Amazon Web Services (AWS); Bellevue (WA); On-site; USD 136,000 - 184,000; Full time

Benefits offered by this job

Health insurance

401(k) matching

Paid time off

Job summary

Amazon Data Services, Inc. in Bellevue, WA seeks an Applied Scientist to advance ML and auditing frameworks for high-quality data powering Nova and LLMs.

You will design quality metrics, manual and automated evaluations, and integrate data collection QA into production workflows. You will collaborate with scientists and engineers to build judge architectures, evaluation rubrics, and automated quality assessment models, ensuring robust data for state-of-the-art AI services and customer

Qualifications

• Master's degree or equivalent in a quantitative field.
• Experience with Java, C++, Python and SQL in ML/DS contexts.

Responsibilities

• Collaborate with core scientist teams to develop and implement comprehensive quality strategies and auditing frameworks for data collection workflows.
• Design auditing strategies with SOPs, quality metrics, and sampling methods for improved model benchmarking.

Skills

Java

C++

Python

Education

Master's degree in computer science, mathematics, statistics, machine learning or equivalent quantitative field

Tools

SQL

Oracle

Data Warehouse

Job description

Amazon Data Services, Inc. in Bellevue, WA seeks an Applied Scientist to advance ML and auditing frameworks for high-quality data powering Nova and LLMs.

You will design quality metrics, manual and automated evaluations, and integrate data collection QA into production workflows. You will collaborate with scientists and engineers to build judge architectures, evaluation rubrics, and automated quality assessment models, ensuring robust data for state-of-the-art AI services and customer