Applied Scientist, LLM Quality & Evaluation

Socket.dev; Bellevue (WA); On-site; USD 136,000 - 184,000; Full time

Benefits offered by this job

RSUs

Sign-on bonus

Health insurance

Job summary

Amazon is seeking an Applied Scientist for the AGI team to advance data quality for LLMs and multimodal systems, building auditing frameworks and automated evaluation pipelines.

You will design SOPs, governance metrics, root-cause analyses, and mentoring programs to uplift data quality and model performance. Collaboration with engineers and vendor teams will translate research into scalable quality solutions, impacting customer experiences and state-of-the-art AI products.

Qualifications

• Master’s degree or equivalent in a quantitative field.
• Experience programming in Java, C++, Python or related language and SQL.
• Familiarity with RDBMS like Oracle or data warehouse concepts.

Responsibilities

• Lead development of quality strategies and auditing frameworks for data integrity.
• Design auditing SOPs, quality metrics, and sampling methods to improve benchmarks.
• Conduct expert-level audits and coach teams to uplift quality capabilities.
• Develop LLMA-as-a-Judge architectures and automated quality assessment models.

Skills

Java

C++

Python

SQL

Education

Master's degree in computer science, mathematics, statistics, machine learning or equivalent quantitative field

Tools

Oracle

Job description

Amazon is seeking an Applied Scientist for the AGI team to advance data quality for LLMs and multimodal systems, building auditing frameworks and automated evaluation pipelines.

You will design SOPs, governance metrics, root-cause analyses, and mentoring programs to uplift data quality and model performance. Collaboration with engineers and vendor teams will translate research into scalable quality solutions, impacting customer experiences and state-of-the-art AI products.