Applied Scientist, AGI Quality & LLM Evaluation

Amazon; Factoria (WA); On-site; USD 136,000 - 184,000; Full time

Job summary

Amazon's AGI team seeks an Applied Scientist with a strong ML, statistics, and QA background to build high-quality data for LLMs and multimodal systems. You will help craft auditing frameworks and SOPs, ensuring robust data pipelines and evaluation processes.

You will design and implement LLM-as-a-Judge systems, build evaluation rubrics, and develop models for automated quality assessment, while collaborating with engineers, scientists, and vendors to deliver top-tier data products.

Qualifications

• Master's degree in computer science, mathematics, statistics, machine learning or equivalent quantitative field.
• Experience programming in Java, C++, Python or related language.
• Experience with SQL and an RDBMS (e.g., Oracle) or Data Warehouse.

Responsibilities

• Develop quality strategies and auditing frameworks to safeguard data collection workflows.
• Design auditing strategies with SOPs, quality metrics, and sampling methodologies.
• Perform expert-level manual audits and meta-audits to uplift overall quality capabilities.
• Develop and maintain LLM-as-a-Judge systems with evaluation rubrics and ML models.
• Set up data collection workflows configurations and communicate quality feedback to stakeholders.

Skills

Java

C++

Python

SQL

Education

Master's degree in computer science, mathematics, statistics, machine learning or equivalent quantitative field

Tools

Oracle

Data Warehouse

Job description

Amazon's AGI team seeks an Applied Scientist with a strong ML, statistics, and QA background to build high-quality data for LLMs and multimodal systems. You will help craft auditing frameworks and SOPs, ensuring robust data pipelines and evaluation processes.

You will design and implement LLM-as-a-Judge systems, build evaluation rubrics, and develop models for automated quality assessment, while collaborating with engineers, scientists, and vendors to deliver top-tier data products.