This role focuses on leading the evaluation and benchmarking data development that supports Amazon Quick. You will design datasets, define metrics and evaluation frameworks, and use the resulting insights to guide product decisions and model improvements.
Responsibilities
- Design and build evaluation and benchmark datasets.
- Define quality metrics and evaluation frameworks, including human evaluation, model-based evaluation using LLM-as-a-Judge, and checks on the correlation between human and model evaluations.
- Run human annotation programs end to end, including writing guidelines, measuring inter-annotator agreement, and managing data quality controls.
- Analyze model and agent performance to identify gaps, root causes, and the most impactful improvement opportunities.
- Collaborate with science, engineering, and product teams to translate evaluation results into launch decisions and roadmap priorities.
- Present findings and recommendations to senior leadership.
Requirements
- 5+ years of experience with data querying languages (for example, SQL), scripting languages (for example, Python), or statistical and mathematical software (for example, R, SAS, Matlab).
- 4+ years of data scientist experience.
- Experience leading and mentoring on a data science team.
Preferred Qualifications
- 2+ years of experience in data visualization using AWS QuickSight, Tableau, R Shiny, or similar tools.
- Experience managing data pipelines.
- Experience evaluating LLMs, RAG systems, or agentic AI applications.
Technologies
SQL, Python, R, SAS, Matlab, AWS QuickSight, Tableau, R Shiny, LLM-as-a-Judge, RAG
Compensation and Location
Location: Santa Clara, CA (onsite).
Salary: USD 183,000 - 247,600 per year.
Benefits
- Comprehensive benefits including health insurance (medical, dental, vision, prescription, Basic Life & AD&D insurance and option for Supplemental life plans), EAP, mental health support, medical advice line, flexible spending accounts, and adoption and surrogacy reimbursement coverage.
- 401(k) matching.
- Paid time off.
- Parental leave.
- Sign-on payments and restricted stock units (RSUs).