Applied Scientist, AGI & LLM Quality & Evaluation

Amazon; Boston (MA); On-site; USD 136,000 - 184,000; Full time

Benefits offered by this job

Health insurance

RSUs included

401(k) matching

Paid time off

Parental leave

Job summary

Amazon's AGI team is seeking an Applied Scientist to lead quality strategies for data used in LLMs and multimodal systems. You will design auditing frameworks, run expert-level audits, and build automated judging systems to safeguard data integrity and model performance.

You will collaborate with the Nova model team and vendors to optimize data collection, provide quality feedback, and contribute to training data that powers next-generation AI services.

Qualifications

• Master's degree in computer science, mathematics, statistics, machine learning or equivalent quantitative field.
• Experience programming in Java, C++, Python or related language.
• Experience with SQL and an RDBMS (e.g., Oracle) or Data Warehouse.

Responsibilities

• Collaborate with core scientist team developing Amanda Nova models to build quality strategies and auditing frameworks.
• Design auditing strategies with SOPs, quality metrics, and sampling methods to improve Nova benchmarks.
• Perform expert-level manual audits and meta-audits to uplift overall quality capabilities.
• Develop and maintain LLM-as-a-Judge systems, including judge architectures and evaluation rubrics.
• Configure data collection workflows and communicate quality feedback to stakeholders.
• Contribute to high-quality training and evaluation data powering state-of-the-art LLM products.

Skills

Java

C++

Python

SQL

Publications in top venues

Education

Master's degree in computer science, mathematics, statistics, machine learning or equivalent quantitative field

Job description

Amazon's AGI team is seeking an Applied Scientist to lead quality strategies for data used in LLMs and multimodal systems. You will design auditing frameworks, run expert-level audits, and build automated judging systems to safeguard data integrity and model performance.

You will collaborate with the Nova model team and vendors to optimize data collection, provide quality feedback, and contribute to training data that powers next-generation AI services.