Senior Data Scientist: Evaluation & Benchmarking

Amazon Jobs

Santa Clara (CA)

On-site

USD 183,000 - 248,000

Full time

10 days ago

A complete application in a minute

Tailored resume and cover letter, ready to send.

Get past ATS filters

Benefits offered by this job

Health insurance
RSUs
401(k)

Job summary

Amazon Quick’s Data Science team is hiring a Data Scientist II to lead evaluation and benchmarking data development for Quick features. You will design datasets, metrics, and evaluation frameworks to show whether Quick meets customer needs and to drive model improvements.

You will partner with science, engineering, and product teams to turn evaluation results into launch decisions and roadmaps, present findings to senior leadership, and mentor junior teammates in data-driven methods.

Qualifications

  • 5+ years of data querying languages (e.g. SQL), scripting languages (e.g. Python) or statistical/mathematical software (e.g. R, SAS, Matlab, etc.) experience
  • 4+ years of data scientist experience
  • Experience as a leader and mentor on a data science team
  • 2+ years of data visualization using AWS QuickSight, Tableau, R Shiny, etc. experience
  • Experience managing data pipelines
  • Experience evaluating LLMs, RAG systems, or agentic AI applications

Responsibilities

  • Design and build evaluation and benchmark datasets
  • Define quality metrics and evaluation frameworks, including human evaluation, model-based evaluation (LLM-as-a-Judge), and checks on how well the two correlate
  • Run human annotation programs from start to finish: write guidelines, measure inter-annotator agreement, and control data quality
  • Analyze model and agent performance to find gaps, root causes, and the highest-impact improvements
  • Partner with science, engineering, and product teams to turn evaluation results into launch decisions and roadmap priorities
  • Present findings and recommendations to senior leadership

Skills

SQL
Python
Leadership
Data Visualization
Data Pipelines
LLM/RAG Evaluation

Tools

AWS QuickSight
Tableau
R Shiny

Job description

Amazon Quick’s Data Science team is hiring a Data Scientist II to lead evaluation and benchmarking data development for Quick features. You will design datasets, metrics, and evaluation frameworks to show whether Quick meets customer needs and to drive model improvements.

You will partner with science, engineering, and product teams to turn evaluation results into launch decisions and roadmaps, present findings to senior leadership, and mentor junior teammates in data-driven methods.

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Senior Data Scientist — AI Evaluation & Benchmarking
Senior Data Scientist — AI Evaluation & Benchmarking

Amazon Inc. • Omaha (NE)

On-site
USD 183,000 - 248,000
Senior Data Scientist: AI Evaluation & Benchmarking Lead
Senior Data Scientist: AI Evaluation & Benchmarking Lead

Amazon.com Inc. • Santa Clara (CA)

On-site
USD 183,000 - 248,000
Senior Data Scientist - AI Evaluation & Benchmarking
Senior Data Scientist - AI Evaluation & Benchmarking

Amazon Web Services (AWS) • Santa Clara (CA)

On-site
USD 183,000 - 248,000
Health insurance
RSU grants
401(k) matching
+1
Data Scientist II: AI Evaluation & Benchmarking
Data Scientist II: AI Evaluation & Benchmarking

Amazon Inc. • Santa Clara (CA)

Hybrid
USD 140,000 - 200,000
Data Scientist II: GenAI Evaluation & Benchmarking (Hybrid)
Data Scientist II: GenAI Evaluation & Benchmarking (Hybrid)

Amazon.com Inc. • Omaha (NE)

Hybrid
USD 157,000 - 213,000
Data Scientist, AI Evaluation & Benchmarking (Hybrid)
Data Scientist, AI Evaluation & Benchmarking (Hybrid)

Amazon • Seattle (WA)

Hybrid
USD 157,000 - 213,000
Health insurance
Hybrid work model
Data Scientist II — AI Evaluation & Benchmarking (Hybrid)
Data Scientist II — AI Evaluation & Benchmarking (Hybrid)

Amazon Web Services (AWS) • Santa Clara (CA)

Hybrid
USD 157,000 - 213,000
Sr Data Scientist
Sr Data Scientist

Amazon Web Services (AWS) • Santa Clara (CA)

On-site
USD 183,000 - 248,000
Health insurance
RSU grants
401(k) matching
+1
Sr Data Scientist
Sr Data Scientist

Amazon Development Center U.S., Inc. • Santa Clara (CA)

On-site
USD 183,000 - 248,000
Health insurance
401(k) matching
Paid time off
+2
Data Scientist, AWS Quick Data
Data Scientist, AWS Quick Data

Amazon Inc. • Santa Clara (CA)

Hybrid
USD 140,000 - 200,000