Shopify Storefront QA Specialist — Remote Contract

mercor; United States; Remote; USD 55,000 - 69,000; Full time

Job summary

Mercor connects elite creative and technical talent with leading AI research labs. The Shopify Storefront Agent Evaluator role is a remote contract position, paying $40–$50/hour, with up to 5 hours per week.

You will simulate shopper interactions, assess responses for relevance, accuracy, and safety, and provide concise feedback to help the team improve. Ideal candidates have e-commerce fluency, careful judgment, and Shopify experience; strong written English is essential.

Qualifications

• Consumer e-commerce fluency; you shop online regularly and understand effective product recommendations.
• Careful, consistent judgment; apply the rubric uniformly across responses.
• Clear, concise written English; articulate reasons for response failures in a sentence or two.

Responsibilities

• Evaluate the Storefront Agent by simulating shopper interactions, incl product discovery, comparisons, inquiries about sizing, stock, shipping, and returns.
• Grade responses based on Relevance, Accuracy, and Safety to ensure they meet shopper expectations and policies.
• Provide concise feedback on any failures to ensure clarity for team members not present during evaluation.
• Work independently to deliver precise and repeatable judgments on agent performance.

Skills

Consumer e-commerce fluency

Careful, consistent judgment

Clear, concise written English

Job description

Mercor connects elite creative and technical talent with leading AI research labs. The Shopify Storefront Agent Evaluator role is a remote contract position, paying $40–$50/hour, with up to 5 hours per week.

You will simulate shopper interactions, assess responses for relevance, accuracy, and safety, and provide concise feedback to help the team improve. Ideal candidates have e-commerce fluency, careful judgment, and Shopify experience; strong written English is essential.