Applied Scientist, AGI Quality & LLM Evaluation

Amazon.com Services LLC; Bellevue (WA); On-site; USD 136,000 - 184,000; Full time

Benefits offered by this job

Health insurance

401(k) matching

Paid time off

Parental leave

RSUs

Job summary

Amazon.com Services LLC is seeking an Applied Scientist for the AGI team to build quality strategies, auditing frameworks, and automated evaluation systems for LLMs and multimodal data. You will design judge architectures, develop evaluation rubrics, and improve data quality powering state-of-the-art models.

You will collaborate with engineers and scientists on data collection workflows, conduct audits, and coach teams to uplift quality standards, impacting customer experiences through superior

Qualifications

• Master's degree in computer science, mathematics, statistics, machine learning or equivalent quantitative field.
• Experience programming in Java, C++, Python or related language.
• Experience with SQL and an RDBMS (e.g., Oracle) or Data Warehouse.

Responsibilities

• Collaborate with core scientist teams developing Amazon Nova models.
• Lead development of quality strategies and auditing frameworks for data collection workflows.
• Design auditing strategies with SOPs, quality metrics and sampling methodologies to improve Nova benchmarks.
• Perform expert-level manual audits and meta-audits to evaluate auditor performance.
• Develop and maintain LLM-as-a-Judge systems, including judge architectures and evaluation rubrics.

Skills

Machine learning

Statistics

Quality assurance

Auditing methodologies

Automated evaluation

Java

C++

Python

SQL

Education

Master's degree in computer science / mathematics / statistics / ML

Tools

Oracle

Job description

Amazon.com Services LLC is seeking an Applied Scientist for the AGI team to build quality strategies, auditing frameworks, and automated evaluation systems for LLMs and multimodal data. You will design judge architectures, develop evaluation rubrics, and improve data quality powering state-of-the-art models.

You will collaborate with engineers and scientists on data collection workflows, conduct audits, and coach teams to uplift quality standards, impacting customer experiences through superior