Data Scientist, AI Evaluation & Benchmarking (Hybrid)

Amazon

Seattle (WA)

Hybrid

USD 157,000 - 213,000

Full time

2 days ago
Be an early applicant

A complete application in a minute

Tailored resume and cover letter, ready to send.

Get past ATS filters

Benefits offered by this job

Health insurance
Hybrid work model

Job summary

Amazon.com Services LLC is seeking a data scientist to drive evaluation and benchmarking for Quick Suite AI features. You will design datasets, generate synthetic data with LLMs, and lead model evaluation audits.

You will collaborate with ML scientists to translate insights into product improvements, build scalable data pipelines, and contribute to Responsible AI initiatives within a hybrid work model across U.S. offices.

Qualifications

  • 2+ years of data scientist experience
  • 3+ years of data querying languages (SQL) or scripting languages (Python)
  • 3+ years of ML/statistical modeling data analysis tools and techniques
  • 1+ years working with AI systems
  • 1+ years creating educational content
  • Master's degree in STEM or equivalent experience
  • Experience applying theoretical models in applied environments

Responsibilities

  • Design and develop evaluation and benchmarking datasets for Quick Suite AI-powered features
  • Leverage LLMs for synthetic data generation and data evaluation
  • Create ground truth datasets with QA pairs across domains
  • Lead human annotation initiatives and model evaluation audits
  • Develop annotation guidelines and quality frameworks
  • Conduct statistical analysis to measure model performance
  • Collaborate with ML scientists to translate insights into product improvements
  • Build scalable data pipelines to support evaluation
  • Contribute to Responsible AI initiatives with safety and fairness datasets

Job description

Amazon.com Services LLC is seeking a data scientist to drive evaluation and benchmarking for Quick Suite AI features. You will design datasets, generate synthetic data with LLMs, and lead model evaluation audits.

You will collaborate with ML scientists to translate insights into product improvements, build scalable data pipelines, and contribute to Responsible AI initiatives within a hybrid work model across U.S. offices.

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Data Scientist II: GenAI Evaluation & Benchmarking (Hybrid)
Data Scientist II: GenAI Evaluation & Benchmarking (Hybrid)

Amazon.com Inc. • Omaha (NE)

Hybrid
USD 157,000 - 213,000
Data Scientist II — AI Evaluation & Benchmarking (Hybrid)
Data Scientist II — AI Evaluation & Benchmarking (Hybrid)

Amazon Web Services (AWS) • Santa Clara (CA)

Hybrid
USD 157,000 - 213,000
Data Scientist II: AI Evaluation & Benchmarking
Data Scientist II: AI Evaluation & Benchmarking

Amazon Inc. • Santa Clara (CA)

Hybrid
USD 140,000 - 200,000
Senior Data Scientist — AI Evaluation & Benchmarking
Senior Data Scientist — AI Evaluation & Benchmarking

Amazon Inc. • Omaha (NE)

On-site
USD 183,000 - 248,000
Senior Data Scientist: AI Evaluation & Benchmarking Lead
Senior Data Scientist: AI Evaluation & Benchmarking Lead

Amazon.com Inc. • Santa Clara (CA)

On-site
USD 183,000 - 248,000
Senior Data Scientist - AI Evaluation & Benchmarking
Senior Data Scientist - AI Evaluation & Benchmarking

Amazon Web Services (AWS) • Santa Clara (CA)

On-site
USD 183,000 - 248,000
Health insurance
RSU grants
401(k) matching
+1
Senior Data Scientist: Evaluation & Benchmarking
Senior Data Scientist: Evaluation & Benchmarking

Amazon Jobs • Santa Clara (CA)

On-site
USD 183,000 - 248,000
Health insurance
RSUs
401(k)
Data Scientist, AWS Quick Data
Data Scientist, AWS Quick Data

Amazon Inc. • Santa Clara (CA)

Hybrid
USD 140,000 - 200,000
Senior Data Scientist - Evaluation & LLM Benchmarking Lead
Senior Data Scientist - Evaluation & LLM Benchmarking Lead

Amazon Development Center U.S., Inc. • Santa Clara (CA)

On-site
USD 183,000 - 248,000
Health insurance
401(k) matching
Paid time off
+2
Data Scientist, AWS Quick Data
Data Scientist, AWS Quick Data

Amazon Web Services (AWS) • Santa Clara (CA)

Hybrid
USD 157,000 - 213,000