Content Risk Analyst, ASCI

Amazon; Bengaluru; On-site; INR 1,200,000 - 1,800,000; Full time

Job summary

Amazon in Bengaluru seeks a Sr. Prod Compliance Associate to drive AI model improvement, red teaming, and customer feedback triage across Alexa experiences. You will leverage AI tools while following strict security protocols in restricted environments.

You will collaborate with AI automation, maintain SOPs, draft audit reports, and mentor peers. This role covers Alexa, Ring, Automotive, and Kids experiences with focus on safety, accuracy, and policy compliance.

Qualifications

• Bachelor's degree required.
• 1+ years of experience in content moderation, data annotation, AI/ML operations, or compliance operations.
• Ability to follow SOPs and apply judgment in a fast-changing environment.

Responsibilities

• AI Data Annotation, Quality Auditing & Model Improvement.
• Perform multi-domain data annotation and golden dataset curation for AI model improvement, following established SOPs and quality benchmarks.
• Validate and audit LLM judge outputs through human-in-the-loop review; use AI tools to flag patterns and anomalies.
• Maintain annotation accuracy and audit throughput targets; use AI-assisted error analysis to identify systematic failures and elevate to Process Experts.
• Participate in calibration exercises to ensure reviewer consistency and draft audit reports for program leads.
• Red Teaming Execution across Alexa experiences (Alexa+, Ring, Automotive, Kids).
• Identify, document, and categorize policy violations, safety gaps, and vulnerabilities by severity and impact.
• Draft red teaming documentation and vulnerability summaries, ensuring findings are clearly categorized for program leads.
• Customer Feedback Triage & Analysis: classify and triage daily customer feedback records per SOPs, surface emerging patterns.
• Identify recurring issues, escalation-worthy trends, and edge cases in customer interaction data.
• Handle complex cases requiring human judgment alongside AI-automated pre-filtering and routing.
• Generate trend summaries and insight reports for program leads.
• Incident Monitoring & Escalation Support: monitor for policy violations and emerging issues, draft escalation tickets and incident summaries.
• Coordinate with Operations, Engineering, and Science teams to drive timely resolution; elevate complex cases.

Skills

SOP adherence

AI/ML basics

Content moderation

Analytical thinking

Education

Bachelor's degree

Tools

AI annotation tools

LLM evaluation

Data analysis tools

Job description

Description

As a Sr. Prod Compliance Associate in the ASCI team, you will execute AI model improvement, red teaming, customer feedback triage, and incident monitoring tasks across Alexa experiences. You will apply judgment within established SOPs and use AI tools to support your work across non-sensitive workstreams. You will work alongside AI automation (LLM judges, AI agents) as a human-in-the-loop to ensure quality and safety standards are met. Associates are expected to use AI-powered tools (e.g., LLM assistants, summarizers, pattern analysis tools) as a default in all workstreams except when handling customer-sensitive data, which must follow security protocols in the restricted environment.

Key job responsibilities

• AI Data Annotation, Quality Auditing & Model Improvement
• Perform multi-domain data annotation and golden dataset curation for AI model improvement, following established SOPs and quality benchmarks
• Validate and audit LLM judge outputs through human-in-the-loop review; use AI tools to flag patterns and anomalies
• Maintain annotation accuracy and audit throughput targets; use AI-assisted error analysis to identify systematic failures and elevate to Process Experts
• Participate in calibration exercises to ensure reviewer consistency and draft audit reports for program leads
• Red Teaming Execution
• Execute adversarial testing across multiple Alexa experiences (Alexa+, Ring, Automotive, Kids), using AI to generate diverse attack vectors and edge cases
• Identify, document, and categorize policy violations, safety gaps, and vulnerabilities by severity and impact
• Draft red teaming documentation and vulnerability summaries, ensuring findings are clearly categorized for program leads
• Customer Feedback Triage & Analysis
• Classify and triage daily customer feedback records per SOPs, using AI to pre-sort and surface emerging patterns
• Identify recurring issues, escalation-worthy trends, and edge cases in customer interaction data
• Handle complex cases requiring human judgment alongside AI-automated pre-filtering and routing
• Generate trend summaries and insight reports for program leads
• Incident Monitoring & Escalation Support
• Monitor for policy violations and emerging issues per established protocols, using AI to track incident patterns across workstreams
• Draft escalation tickets, incident summaries, and root-cause hypotheses to support cross-functional response
• Coordinate with Operations, Engineering, and Science teams to drive timely resolution; elevate complex cases.
• Contribute observations and incident learnings to refine SOPs, tools, and response playbooks
• Process Improvement & Team Support
• Identify and propose process improvements to enhance operational efficiency, using AI to prototype automation and workflow optimizations
• Draft and maintain SOPs, training materials, and onboarding guides using AI-assisted documentation tools
• Mentor junior associates and peers; share learnings and conduct training sessions
• Contribute to UAT for new tools (LLM judge workflows, AI agent pipelines) and recommend adoption to program leads

A day in the life

You check incident alerts and prioritize escalations, then move into your core work block, validating LLM judge outputs, curating datasets, or reviewing flagged discrepancies for accuracy. You also handle customer feedback triage, managing the ambiguous cases and edge cases that AI pre-sorting can't confidently classify from the day's records. Other responsibilities rotate between red teaming, drafting incident summaries, calibration sessions, or process improvements like updating SOPs and mentoring junior associates. You wrap up by logging throughput and flagging open items for the next shift.

About The Team

The ASCI (AI Safety, Compliance & Improvement) Programs & Operations team works at the intersection of AI quality and customer safety across Alexa experiences. Our Model Improvement & RAI CX Ops group is responsible for ensuring AI systems meet safety, accuracy, and policy standards through human-in-the-loop review, red teaming, and feedback analysis. We operate across Alexa+, Ring, Automotive, and Kids experiences, combining human judgment with AI-powered tools to keep pace with rapidly evolving models and customer needs.

Basic Qualifications

• Bachelor's degree
• Experience in Artificial Intelligence/Machine Learning (AI/ML) Standards Law
• 1+ years of experience in content moderation, data annotation, AI/ML operations, or compliance operations
• Ability to follow SOPs and apply judgment in a fast-changing environment

Preferred Qualifications

• Experience working with Data & AI related technologies, including, but not limited to, AI/ML, GenAI, Analytics, Database, and/or Storage
• Experience with AI annotation tools, LLM outputs, or model evaluation frameworks
• Familiarity with red teaming, content safety, or responsible AI operations
• Exposure to incident management or escalation handling workflows

Our inclusive culture empowers Amazonians to deliver the best results for our customers. If you have a disability and need a workplace accommodation or adjustment during the application and hiring process, including support for the interview or onboarding process, please visit https://amazon.jobs/content/en/how-we-hire/accommodations for more information. If the country/region you’re applying in isn’t listed, please contact your Recruiting Partner.

Company - ADCI - BLR 14 SEZ

Job ID: A10480093