E-commerce Annotation Specialist: AI Shopping Feedback

Mercor; Los Angeles (CA); On-site; USD 34,000 - 48,000; Full time

Job summary

Shopify’s Shop App Agent is the AI assistant buyers talk to inside the Shop app — product discovery and search, orders, shipping and tracking, account questions, and Shop Cash.

Your job is to read a full shopper conversation, identify the exact thumbs-down turn, classify it with one primary domain tag and one secondary tag, and write a short comment explaining what the shopper wanted, what the agent did or failed to do, and why the labels fit. Commitment: ~5 hours/week.

Qualifications

• Ability to apply a fixed taxonomy across conversations.
• Comfort reading structured traces: tool calls and outputs.
• Restraint under ambiguity: label unknowns honestly.
• Clear, concise written English.

Responsibilities

• Read a full shopper conversation including tool calls and responses.
• Identify the turn that received the thumbs-down.
• Classify with a primary tag and a secondary tag.
• Write a short comment explaining the needs, actions, and rationale.

Skills

Rule discipline

Structured traces

Ambiguity restraint

Clear written English

Consumer e-commerce fluency

Annotation/model-evaluation experience

Shop app familiarity

Support operations familiarity

Job description

Shopify’s Shop App Agent is the AI assistant buyers talk to inside the Shop app — product discovery and search, orders, shipping and tracking, account questions, and Shop Cash.

Your job is to read a full shopper conversation, identify the exact thumbs-down turn, classify it with one primary domain tag and one secondary tag, and write a short comment explaining what the shopper wanted, what the agent did or failed to do, and why the labels fit. Commitment: ~5 hours/week.