Applied Scientist - AGI Quality & LLM Evaluation

Socket.dev; Bellevue (WA); On-site; USD 136,000 - 184,000; Full time

Benefits offered by this job

Health insurance

RSUs

Sign-on bonus

Parental leave

Job summary

Amazon's Artificial General Intelligence (AGI) team is seeking an Applied Scientist with a strong background in machine learning, statistics, quality assurance, auditing methodologies, and automated evaluation systems to ensure the highest data quality for LLMs and multimodal systems.

You will design auditing strategies, create SOPs, quality metrics, and sampling methods, and build LLM-as-a-Judge architectures with evaluation rubrics and automated models.

Qualifications

• Master's degree in computer science, mathematics, statistics, machine learning or equivalent quantitative field.
• Experience programming in Java, C++, Python or related language.
• Experience with SQL and an RDBMS (e.g., Oracle) or Data Warehouse.

Responsibilities

• Design auditing strategies with SOPs, quality metrics and sampling methodologies.
• Develop and maintain LLM-as-a-Judge systems and evaluation rubrics.
• Perform expert-level audits, meta-audits, and coaching to uplift quality capabilities.
• Collaborate with engineers and domain experts to implement quality workflows.

Skills

ML experience

Statistics

Quality assurance

Education

Master's degree in CS/Math/Stats/ML

Tools

SQL

RDBMS

Data Warehouse

Job description

Amazon's Artificial General Intelligence (AGI) team is seeking an Applied Scientist with a strong background in machine learning, statistics, quality assurance, auditing methodologies, and automated evaluation systems to ensure the highest data quality for LLMs and multimodal systems.

You will design auditing strategies, create SOPs, quality metrics, and sampling methods, and build LLM-as-a-Judge architectures with evaluation rubrics and automated models.