Applied Scientist, LLM Quality & Auditing

Amazon; Bellevue (WA); Hybrid; USD 136,000 - 184,000; Full time

Benefits offered by this job

Health insurance

RSUs

401(k) matching

Paid time off

Parental leave

Job summary

Amazon Data Services, Inc. seeks an Applied Scientist with a strong ML and statistics background to build auditing systems for data quality. You will lead quality strategies, design framework SOPs, and develop automated evaluation for LLM-based judging.

Collaboration with scientists, engineers, and vendor teams is essential to improve Nova benchmark performance. The role involves audits, coaching, and implementing data-collection quality practices that directly impact customer experiences with

Qualifications

• Master's degree in CS, math, stats, ML or equivalent.
• Experience programming in Java, C++, Python.
• Experience with SQL and an RDBMS or Data Warehouse.

Responsibilities

• Develop quality strategies and auditing frameworks for data collection workflows.
• Design LLM-as-a-Judge systems and evaluation rubrics.
• Collaborate with engineers and vendors to implement quality solutions.
• Perform expert-level audits and coach teams to improve quality.

Education

Master's degree in computer science, mathematics, statistics, machine learning or equivalent quantitative field

Tools

Java

C++

Python

SQL and RDBMS/Data Warehouse

Job description

Amazon Data Services, Inc. seeks an Applied Scientist with a strong ML and statistics background to build auditing systems for data quality. You will lead quality strategies, design framework SOPs, and develop automated evaluation for LLM-based judging.

Collaboration with scientists, engineers, and vendor teams is essential to improve Nova benchmark performance. The role involves audits, coaching, and implementing data-collection quality practices that directly impact customer experiences with