Applied Scientist - AGI & LLM Quality & Evaluation

Amazon Science; Bellevue (WA); On-site; USD 136,000 - 184,000; Full time

Job summary

Amazon Science is seeking an Applied Scientist to lead quality auditing and evaluation for AGI initiatives, focusing on data integrity for LLMs and multimodal systems. You will design auditing frameworks, SOPs, metrics, and evaluation rubrics, while mentoring teams and building automated quality assessment tools.

The role involves collaboration with core scientists and engineers to enhance data quality, contribute to high-stakes benchmarks, and influence customer experiences through robust data

Qualifications

• Master's degree in computer science, mathematics, statistics, machine learning or equivalent.
• Experience programming in Java, C++, Python or related language.
• Experience with SQL and an RDBMS (e.g., Oracle) or Data Warehouse.

Responsibilities

• Develop quality strategies and auditing frameworks for data collection workflows.
• Design auditing strategies with SOPs, metrics, and sampling methodologies.
• Build and maintain LLM-as-a-Judge systems for automated quality assessment.
• Collaborate with engineers and vendors to implement quality solutions.

Skills

Java

C++

Python

Education

Master's degree in computer science, mathematics, statistics, machine learning or equivalent

Tools

SQL & RDBMS (Oracle)

Job description

Amazon Science is seeking an Applied Scientist to lead quality auditing and evaluation for AGI initiatives, focusing on data integrity for LLMs and multimodal systems. You will design auditing frameworks, SOPs, metrics, and evaluation rubrics, while mentoring teams and building automated quality assessment tools.

The role involves collaboration with core scientists and engineers to enhance data quality, contribute to high-stakes benchmarks, and influence customer experiences through robust data