Data Scientist II: AI Evaluation & Benchmarking

Amazon Inc.

Santa Clara (CA)

Hybrid

USD 140,000 - 200,000

Full time

10 days ago

Get a reply from this employer

Your resume and cover letter, tailored to exactly what they’re hiring for.

Get past ATS filters

Job summary

Amazon is seeking a Data Scientist II for the Quick Data team in Santa Clara. You will design and implement evaluation datasets, synthesize data with LLMs, and create ground truth QA pairs across domains to benchmark Quick Suite features.

You will lead annotation and model evaluation audits, develop quality guidelines, and perform statistical analyses to drive improvements, partnering with ML engineers and scientists in a hybrid work environment.

Qualifications

  • 2+ years of data scientist experience.
  • 3+ years of SQL, Python, or statistical software experience.
  • 3+ years of machine learning/statistical modeling and data analysis experience.
  • 1+ years of working with or evaluating AI systems.
  • 1+ years of creating or contributing to mathematical textbooks, research papers, or educational content.
  • Master's degree in Science, Technology, Engineering, or Mathematics (STEM).

Responsibilities

  • Design and develop evaluation and benchmarking datasets for Quick Suite AI-powered features.
  • Leverage LLMs for synthetic data generation and data evaluation.
  • Create ground truth datasets with QA pairs across diverse domains.
  • Lead human annotation initiatives and model evaluation audits.
  • Develop annotation guidelines and quality frameworks for evaluation tasks.
  • Conduct statistical analysis to measure model performance and identify failure patterns.
  • Collaborate with ML scientists and engineers to translate evaluation insights into product improvements.
  • Build scalable data pipelines and tools to support continuous evaluation and benchmarking.
  • Contribute to Responsible AI initiatives by developing safety and fairness evaluation datasets.

Skills

Data science

Education

Master's degree in STEM

Tools

SQL
Python
R

Job description

Amazon is seeking a Data Scientist II for the Quick Data team in Santa Clara. You will design and implement evaluation datasets, synthesize data with LLMs, and create ground truth QA pairs across domains to benchmark Quick Suite features.

You will lead annotation and model evaluation audits, develop quality guidelines, and perform statistical analyses to drive improvements, partnering with ML engineers and scientists in a hybrid work environment.

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Data Scientist II: GenAI Evaluation & Benchmarking (Hybrid)
Data Scientist II: GenAI Evaluation & Benchmarking (Hybrid)

Amazon.com Inc. • Omaha (NE)

Hybrid
USD 157,000 - 213,000
Senior Data Scientist: AI Evaluation & Benchmarking Lead
Senior Data Scientist: AI Evaluation & Benchmarking Lead

Amazon.com Inc. • Santa Clara (CA)

On-site
USD 183,000 - 248,000
Data Scientist II — AI Evaluation & Benchmarking (Hybrid)
Data Scientist II — AI Evaluation & Benchmarking (Hybrid)

Amazon Web Services (AWS) • Santa Clara (CA)

Hybrid
USD 157,000 - 213,000
Senior Data Scientist - AI Evaluation & Benchmarking
Senior Data Scientist - AI Evaluation & Benchmarking

Amazon Web Services (AWS) • Santa Clara (CA)

On-site
USD 183,000 - 248,000
Health insurance
RSU grants
401(k) matching
+1
Data Scientist, AI Evaluation & Benchmarking (Hybrid)
Data Scientist, AI Evaluation & Benchmarking (Hybrid)

Amazon • Seattle (WA)

Hybrid
USD 157,000 - 213,000
Health insurance
Hybrid work model
Senior Data Scientist — AI Evaluation & Benchmarking
Senior Data Scientist — AI Evaluation & Benchmarking

Amazon Inc. • Omaha (NE)

On-site
USD 183,000 - 248,000
Senior Data Scientist: Evaluation & Benchmarking
Senior Data Scientist: Evaluation & Benchmarking

Amazon Jobs • Santa Clara (CA)

On-site
USD 183,000 - 248,000
Health insurance
RSUs
401(k)
Data Scientist, AWS Quick Data
Data Scientist, AWS Quick Data

Amazon Inc. • Santa Clara (CA)

Hybrid
USD 140,000 - 200,000
Sr Data Scientist
Sr Data Scientist

Amazon Web Services (AWS) • Santa Clara (CA)

On-site
USD 183,000 - 248,000
Health insurance
RSU grants
401(k) matching
+1
Data Scientist, AWS Quick Data
Data Scientist, AWS Quick Data

Amazon Web Services (AWS) • Santa Clara (CA)

Hybrid
USD 157,000 - 213,000