Data Scientist II — AI Evaluation & Benchmarking (Hybrid)

Amazon Web Services (AWS)

Santa Clara (CA)

Hybrid

USD 157,000 - 213,000

Full time

3 days ago
Be an early applicant

A complete application in a minute

Tailored resume and cover letter, ready to send.

Get past ATS filters

Job summary

Amazon.com Services LLC in California is hiring a Data Scientist II for the Quick Data team to develop evaluation and benchmarking datasets for AI-powered features. You will work on human evaluations, Responsible AI safeguards, and retrieval-augmented generation, ensuring enterprise-ready data quality and safe AI practices.

You will design experiments to assess data impact on model performance, use LLMs for synthetic data, and collaborate with ML engineers to translate findings into product

Qualifications

  • 2+ years of data scientist experience.
  • 3+ years of data querying languages (e.g. SQL) experience.
  • 3+ years of machine learning/statistical modeling data analysis tools and techniques.
  • 1+ years of working with or evaluating AI systems.
  • 1+ years of creating or contributing to mathematical textbooks, research papers, or educational content.
  • Master's degree in STEM or equivalent experience.
  • Experience applying theoretical models in an applied environment.

Responsibilities

  • Design and develop comprehensive evaluation and benchmarking datasets for Quick Suite AI-powered features.
  • Leverage LLMs for synthetic data corpora generation; data evaluation and quality assessment using LLM-as-a-judge settings.
  • Create ground truth datasets with high-quality question-answer pairs across diverse domains and use cases.
  • Lead human annotation initiatives and model evaluation audits to ensure data quality and relevance.
  • Develop and refine annotation guidelines and quality frameworks for evaluation tasks.
  • Conduct statistical analysis to measure model performance, identify failure patterns, and guide improvement strategies.
  • Collaborate with ML scientists and engineers to translate evaluation insights into product improvements.
  • Build scalable data pipelines and tools to support continuous evaluation and benchmarking efforts.
  • Contribute to Responsible AI initiatives by developing safety and fairness evaluation datasets.

Skills

SQL
Python
R
Statistics

Education

Master's degree in STEM
PhD in STEM

Tools

SQL
Python
R
SAS
Matlab

Job description

Amazon.com Services LLC in California is hiring a Data Scientist II for the Quick Data team to develop evaluation and benchmarking datasets for AI-powered features. You will work on human evaluations, Responsible AI safeguards, and retrieval-augmented generation, ensuring enterprise-ready data quality and safe AI practices.

You will design experiments to assess data impact on model performance, use LLMs for synthetic data, and collaborate with ML engineers to translate findings into product

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Data Scientist II: GenAI Evaluation & Benchmarking (Hybrid)
Data Scientist II: GenAI Evaluation & Benchmarking (Hybrid)

Amazon.com Inc. • Omaha (NE)

Hybrid
USD 157,000 - 213,000
Data Scientist, AI Evaluation & Benchmarking (Hybrid)
Data Scientist, AI Evaluation & Benchmarking (Hybrid)

Amazon • Seattle (WA)

Hybrid
USD 157,000 - 213,000
Health insurance
Hybrid work model
Data Scientist II: AI Evaluation & Benchmarking
Data Scientist II: AI Evaluation & Benchmarking

Amazon Inc. • Santa Clara (CA)

Hybrid
USD 140,000 - 200,000
Senior Data Scientist — AI Evaluation & Benchmarking
Senior Data Scientist — AI Evaluation & Benchmarking

Amazon Inc. • Omaha (NE)

On-site
USD 183,000 - 248,000
Senior Data Scientist - AI Evaluation & Benchmarking
Senior Data Scientist - AI Evaluation & Benchmarking

Amazon Web Services (AWS) • Santa Clara (CA)

On-site
USD 183,000 - 248,000
Health insurance
RSU grants
401(k) matching
+1
Senior Data Scientist: AI Evaluation & Benchmarking Lead
Senior Data Scientist: AI Evaluation & Benchmarking Lead

Amazon.com Inc. • Santa Clara (CA)

On-site
USD 183,000 - 248,000
Senior Data Scientist: Evaluation & Benchmarking
Senior Data Scientist: Evaluation & Benchmarking

Amazon Jobs • Santa Clara (CA)

On-site
USD 183,000 - 248,000
Health insurance
RSUs
401(k)
Data Scientist, AWS Quick Data
Data Scientist, AWS Quick Data

Amazon Inc. • Santa Clara (CA)

Hybrid
USD 140,000 - 200,000
Data Scientist, AWS Quick Data
Data Scientist, AWS Quick Data

Amazon Web Services (AWS) • Santa Clara (CA)

Hybrid
USD 157,000 - 213,000
Senior Data Scientist - Evaluation & LLM Benchmarking Lead
Senior Data Scientist - Evaluation & LLM Benchmarking Lead

Amazon Development Center U.S., Inc. • Santa Clara (CA)

On-site
USD 183,000 - 248,000
Health insurance
401(k) matching
Paid time off
+2