E-commerce Annotation Specialist - AI Trainer

Mercor; Los Angeles (CA); On-site; USD 34,000 - 48,000; Full time

Job summary

Shopify’s Shop App Agent is the AI assistant buyers talk to inside the Shop app — product discovery and search, orders, shipping and tracking, account questions, and Shop Cash.

Your job is to read a full shopper conversation, identify the exact thumbs-down turn, classify it with one primary domain tag and one secondary tag, and write a short comment explaining what the shopper wanted, what the agent did or failed to do, and why the labels fit. Commitment: ~5 hours/week.

Qualifications

• Ability to apply a fixed taxonomy across conversations.
• Comfort reading structured traces: tool calls and outputs.
• Restraint under ambiguity: label unknowns honestly.
• Clear, concise written English.

Responsibilities

• Read a full shopper conversation including tool calls and responses.
• Identify the turn that received the thumbs-down.
• Classify with a primary tag and a secondary tag.
• Write a short comment explaining the needs, actions, and rationale.

Skills

Rule discipline

Structured traces

Ambiguity restraint

Clear written English

Consumer e-commerce fluency

Annotation/model-evaluation experience

Shop app familiarity

Support operations familiarity

Job description

About the role

Shopify's Shop App Agent is the AI assistant buyers talk to inside the Shop app — product discovery and search, orders, shipping and tracking, account questions, and Shop Cash.

When a shopper is unhappy with one of its answers, they tap thumbs-down — and that's all they leave. No comment, no reason, no survey. Your job is to work out why, from the transcript alone.

What you'll do

• Read a full shopper agent conversation, including the tool calls the agent made and what came back

• Find the specific turn that received the thumbs-down

• Classify it with one primary tag — the domain of the complaint: a disliked recommendation, a failed order lookup, an action the agent couldn't take, a rejection of AI itself, or a genuinely broken response — and one specific secondary tag within it

• Write a short comment: what the shopper wanted, what the agent did or failed to do, and why those tags fit

• Commitment: ~5 hours/week

What makes someone good at this

The hardest part is not guessing. Plenty of thumbs-downs have no visible cause — the agent did nothing wrong, or the shopper simply didn't like the answer. There is an explicit unknown label for exactly that, and using it honestly matters more than producing a confident‑sounding reason. We would rather record "we don't know why" than invent a cause that sends the wrong signal to the model.

Qualifications

Required:

• Rule discipline — you can apply a fixed taxonomy the same way across hundreds of conversations, and you notice when a case sits between two labels rather than forcing it.

• Comfort reading structured traces — tool calls, their inputs, and their raw outputs, so you can tell what the agent actually did from what it merely claimed.

• Restraint under ambiguity — you are willing to label something unknown rather than reach for a plausible‑sounding cause.

• Clear, concise written English — enough to explain in two or three sentences why a conversation got the labels you gave it.

Strongly preferred:

• Consumer e-commerce fluency — orders, tracking, returns, refunds, and a working sense of where a merchant's responsibility ends and the platform's begins.

• Prior annotation, labelling, or model‑evaluation work against a defined rubric or taxonomy.

Preferred (nice to have — we'll ramp you on the specifics):

• Hands‑on experience with the Shop app or other AI shopping assistants as a shopper.

• Familiarity with customer‑support operations and escalation paths.

What this is not

This is not a customer-support role and not an engineering role. You are not fixing the agent and not replying to shoppers — you are diagnosing why a real shopper was unhappy, precisely and repeatably, so the team can measure where the agent falls short.