Applied Scientist, AGI Quality & LLM Evaluation

Amazon Science; Bellevue (WA); On-site; USD 136,000 - 184,000; Full time

Benefits offered by this job

Health insurance

401(k) matching

RSUs

Job summary

Amazon Science is seeking an Applied Scientist to advance data quality for AGI initiatives. You will build auditing frameworks, contribute to the development of evaluation systems, and collaborate with scientist teams to ensure high-quality data powering LLMs and multimodal models.

The role emphasizes designing rigorous QA processes, meta-audits, and coaching to uplift overall quality capabilities, while integrating automated judging systems into data workflows.

Qualifications

• Master's degree in computer science, mathematics, statistics, machine learning or equivalent quantitative field.
• Experience programming in Java, C++, Python or related language.
• Experience with SQL and an RDBMS (e.g., Oracle) or Data Warehouse.

Responsibilities

• Collaborate with core scientist teams to develop quality strategies and auditing frameworks for data collection workflows.
• Design auditing strategies with SOPs, quality metrics, and sampling methodologies to improve model benchmarks.
• Develop and maintain LLM-as-a-Judge systems and automated quality assessment models.

Skills

Java/C++/Python

SQL & Data Warehouse

Education

Master's degree in CS/Math/Stats/ML

Job description

Amazon Science is seeking an Applied Scientist to advance data quality for AGI initiatives. You will build auditing frameworks, contribute to the development of evaluation systems, and collaborate with scientist teams to ensure high-quality data powering LLMs and multimodal models.

The role emphasizes designing rigorous QA processes, meta-audits, and coaching to uplift overall quality capabilities, while integrating automated judging systems into data workflows.