Applied Scientist, AGI Data Quality & LLM Evaluation

Amazon; Boston (MA); On-site; USD 136,000 - 184,000; Full time

Benefits offered by this job

Health insurance

401(k) matching

Paid time off

Parental leave

Job summary

Amazon's AGI team seeks an Applied Scientist to lead quality assurance for data used in training and evaluating large language and multimodal models. You will design auditing frameworks, SOPs, quality metrics, and automated evaluation systems including LLM-as-a-Judge components.

Collaborating with scientists and engineers, you will implement data collection workflows, provide quality feedback to stakeholders, and help power state-of-the-art AI products with high-quality training data.

Qualifications

• Master's degree in computer science, mathematics, statistics, machine learning or equivalent quantitative field.
• Experience programming in Java, C++, Python or related language.
• Experience with SQL and an RDBMS (e.g., Oracle) or Data Warehouse.

Responsibilities

• Lead development of quality strategies and auditing frameworks for data used in training and evaluating LLMs and multimodal models.
• Design SOPs, quality metrics, sampling methods, and automated evaluation systems including LLM-as-a-Judge components.
• Perform expert audits, meta-audits, and coach teams to uplift data quality capabilities.
• Set up data collection workflows and communicate quality feedback to stakeholders.

Tools

Java

C++

Python

SQL

Oracle

Job description

Amazon's AGI team seeks an Applied Scientist to lead quality assurance for data used in training and evaluating large language and multimodal models. You will design auditing frameworks, SOPs, quality metrics, and automated evaluation systems including LLM-as-a-Judge components.

Collaborating with scientists and engineers, you will implement data collection workflows, provide quality feedback to stakeholders, and help power state-of-the-art AI products with high-quality training data.