Senior Data Scientist - Evaluation & LLM Benchmarking Lead

Amazon Development Center U.S., Inc.

Santa Clara (CA)

On-site

USD 183,000 - 248,000

Full time

9 days ago

Get a reply from this employer

Your resume and cover letter, tailored to exactly what they’re hiring for.

Get past ATS filters

Benefits offered by this job

Health insurance
401(k) matching
Paid time off
Parental leave
Restricted stock units (RSUs)

Job summary

Amazon Development Center U.S., Inc. is seeking a senior data scientist to lead the creation of evaluation benchmarks and datasets. You will define metrics, drive model-aware evaluation, and guide product decisions with data-driven insights.

You will supervise data science work, mentor teams, and collaborate across science, engineering, and product groups to shape roadmaps and priorities. Strong expertise in SQL, Python, and statistical tools is required.

Qualifications

  • 5+ years of experience with SQL, Python, or statistical software (R/SAS/Matlab).
  • 4+ years as a data scientist.
  • Experience leading and mentoring a data science team.

Responsibilities

  • Design and build evaluation and benchmark datasets.
  • Define quality metrics and evaluation frameworks, including human evaluation and model-based evaluation.
  • Run human annotation programs end to end, including guidelines and inter-annotator agreement.
  • Analyze model and agent performance to identify gaps and impact opportunities.
  • Collaborate with science, engineering, and product teams to translate results into launch decisions and roadmap priorities.
  • Present findings and recommendations to senior leadership.

Skills

SQL
Python
R
SAS
Matlab
Leadership

Job description

Amazon Development Center U.S., Inc. is seeking a senior data scientist to lead the creation of evaluation benchmarks and datasets. You will define metrics, drive model-aware evaluation, and guide product decisions with data-driven insights.

You will supervise data science work, mentor teams, and collaborate across science, engineering, and product groups to shape roadmaps and priorities. Strong expertise in SQL, Python, and statistical tools is required.

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Data Scientist, AI Evaluation & Benchmarking (Hybrid)
Data Scientist, AI Evaluation & Benchmarking (Hybrid)

Amazon • Seattle (WA)

Hybrid
USD 157,000 - 213,000
Health insurance
Hybrid work model
Senior Data Scientist: AI Evaluation & Benchmarking Lead
Senior Data Scientist: AI Evaluation & Benchmarking Lead

Amazon.com Inc. • Santa Clara (CA)

On-site
USD 183,000 - 248,000
Senior Data Scientist — AI Evaluation & Benchmarking
Senior Data Scientist — AI Evaluation & Benchmarking

Amazon Inc. • Omaha (NE)

On-site
USD 183,000 - 248,000
Senior Data Scientist: Evaluation & Benchmarking
Senior Data Scientist: Evaluation & Benchmarking

Amazon Jobs • Santa Clara (CA)

On-site
USD 183,000 - 248,000
Health insurance
RSUs
401(k)
Senior Data Scientist - AI Evaluation & Benchmarking
Senior Data Scientist - AI Evaluation & Benchmarking

Amazon Web Services (AWS) • Santa Clara (CA)

On-site
USD 183,000 - 248,000
Health insurance
RSU grants
401(k) matching
+1
Data Scientist II — AI Evaluation & Benchmarking (Hybrid)
Data Scientist II — AI Evaluation & Benchmarking (Hybrid)

Amazon Web Services (AWS) • Santa Clara (CA)

Hybrid
USD 157,000 - 213,000
Data Scientist II: AI Evaluation & Benchmarking
Data Scientist II: AI Evaluation & Benchmarking

Amazon Inc. • Santa Clara (CA)

Hybrid
USD 140,000 - 200,000
Senior Data Engineering Leader, AWS Analytics
Senior Data Engineering Leader, AWS Analytics

Amazon Web Services (AWS) • Seattle (WA)

On-site
USD 202,000 - 274,000
Medical, Dental, Vision Coverage
401(k) Plan
Paid Time Off
Sr Data Scientist
Sr Data Scientist

Amazon Development Center U.S., Inc. • Santa Clara (CA)

On-site
USD 183,000 - 248,000
Health insurance
401(k) matching
Paid time off
+2
Data Scientist II: GenAI Evaluation & Benchmarking (Hybrid)
Data Scientist II: GenAI Evaluation & Benchmarking (Hybrid)

Amazon.com Inc. • Omaha (NE)

Hybrid
USD 157,000 - 213,000