Sr Data Scientist

Amazon Development Center U.S., Inc.

Santa Clara (CA)

On-site

USD 183,000 - 248,000

Full time

7 days ago
Be an early applicant

A complete application in a minute

Tailored resume and cover letter, ready to send.

Get past ATS filters

Benefits offered by this job

Health insurance
401(k) matching
Paid time off
Parental leave
Restricted stock units (RSUs)

Job summary

Amazon Development Center U.S., Inc. is seeking a senior data scientist to lead the creation of evaluation benchmarks and datasets. You will define metrics, drive model-aware evaluation, and guide product decisions with data-driven insights.

You will supervise data science work, mentor teams, and collaborate across science, engineering, and product groups to shape roadmaps and priorities. Strong expertise in SQL, Python, and statistical tools is required.

Qualifications

  • 5+ years of experience with SQL, Python, or statistical software (R/SAS/Matlab).
  • 4+ years as a data scientist.
  • Experience leading and mentoring a data science team.

Responsibilities

  • Design and build evaluation and benchmark datasets.
  • Define quality metrics and evaluation frameworks, including human evaluation and model-based evaluation.
  • Run human annotation programs end to end, including guidelines and inter-annotator agreement.
  • Analyze model and agent performance to identify gaps and impact opportunities.
  • Collaborate with science, engineering, and product teams to translate results into launch decisions and roadmap priorities.
  • Present findings and recommendations to senior leadership.

Skills

SQL
Python
R
SAS
Matlab
Leadership

Job description

This role focuses on leading the evaluation and benchmarking data development that supports Amazon Quick. You will design datasets, define metrics and evaluation frameworks, and use the resulting insights to guide product decisions and model improvements.

Responsibilities
  • Design and build evaluation and benchmark datasets.
  • Define quality metrics and evaluation frameworks, including human evaluation, model-based evaluation using LLM-as-a-Judge, and checks on the correlation between human and model evaluations.
  • Run human annotation programs end to end, including writing guidelines, measuring inter-annotator agreement, and managing data quality controls.
  • Analyze model and agent performance to identify gaps, root causes, and the most impactful improvement opportunities.
  • Collaborate with science, engineering, and product teams to translate evaluation results into launch decisions and roadmap priorities.
  • Present findings and recommendations to senior leadership.
Requirements
  • 5+ years of experience with data querying languages (for example, SQL), scripting languages (for example, Python), or statistical and mathematical software (for example, R, SAS, Matlab).
  • 4+ years of data scientist experience.
  • Experience leading and mentoring on a data science team.
Preferred Qualifications
  • 2+ years of experience in data visualization using AWS QuickSight, Tableau, R Shiny, or similar tools.
  • Experience managing data pipelines.
  • Experience evaluating LLMs, RAG systems, or agentic AI applications.
Technologies

SQL, Python, R, SAS, Matlab, AWS QuickSight, Tableau, R Shiny, LLM-as-a-Judge, RAG

Compensation and Location

Location: Santa Clara, CA (onsite).
Salary: USD 183,000 - 247,600 per year.

Benefits
  • Comprehensive benefits including health insurance (medical, dental, vision, prescription, Basic Life & AD&D insurance and option for Supplemental life plans), EAP, mental health support, medical advice line, flexible spending accounts, and adoption and surrogacy reimbursement coverage.
  • 401(k) matching.
  • Paid time off.
  • Parental leave.
  • Sign-on payments and restricted stock units (RSUs).
Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Sr Data Scientist
Sr Data Scientist

Amazon • Santa Clara (CA)

On-site
USD 183,000 - 248,000
Health insurance
401(k) matching
Paid time off
+1
Sr Data Scientist
Sr Data Scientist

Amazon • Omaha (NE)

On-site
USD 183,000 - 248,000
Sr Data Scientist
Sr Data Scientist

Amazon.com Inc. • Santa Clara (CA)

On-site
USD 183,000 - 248,000
Sr Data Scientist
Sr Data Scientist

Amazon Jobs • Santa Clara (CA)

On-site
USD 183,000 - 248,000
Health insurance
RSUs
401(k)
Sr Data Scientist
Sr Data Scientist

Amazon Web Services (AWS) • Santa Clara (CA)

On-site
USD 183,000 - 248,000
Health insurance
RSU grants
401(k) matching
+1
Data Scientist, AWS Quick Data
Data Scientist, AWS Quick Data

Amazon.com • Santa Clara (CA)

Hybrid
USD 157,000 - 213,000
Health insurance
401(k) matching
Paid time off
Data Scientist, AWS Quick Data
Data Scientist, AWS Quick Data

Amazon Web Services (AWS) • Santa Clara (CA)

Hybrid
USD 157,000 - 213,000
Data Scientist, AWS Quick Data
Data Scientist, AWS Quick Data

Amazon • Seattle (WA)

Hybrid
USD 157,000 - 213,000
Health insurance
Hybrid work model
Data Scientist, AWS Quick Data
Data Scientist, AWS Quick Data

Amazon.com Inc. • Omaha (NE)

Hybrid
USD 157,000 - 213,000
Data Scientist III, Rapid & Rural Logistics (R2L) Science & AI
Data Scientist III, Rapid & Rural Logistics (R2L) Science & AI

Amazon Inc. • Bellevue (WA)

On-site
USD 159,000 - 237,000
Medical, Dental, Vision
PTO
401(k) Plan
+1