Shopify Expert - Fully Remote | Upto $50/hr

mercor; United States; Remote; USD 55,000 - 69,000; Full time

Job summary

Mercor connects elite creative and technical talent with leading AI research labs. The Shopify Storefront Agent Evaluator role is a remote contract position, paying $40–$50/hour, with up to 5 hours per week.

You will simulate shopper interactions, assess responses for relevance, accuracy, and safety, and provide concise feedback to help the team improve. Ideal candidates have e-commerce fluency, careful judgment, and Shopify experience; strong written English is essential.

Qualifications

• Consumer e-commerce fluency; you shop online regularly and understand effective product recommendations.
• Careful, consistent judgment; apply the rubric uniformly across responses.
• Clear, concise written English; articulate reasons for response failures in a sentence or two.

Responsibilities

• Evaluate the Storefront Agent by simulating shopper interactions, incl product discovery, comparisons, inquiries about sizing, stock, shipping, and returns.
• Grade responses based on Relevance, Accuracy, and Safety to ensure they meet shopper expectations and policies.
• Provide concise feedback on any failures to ensure clarity for team members not present during evaluation.
• Work independently to deliver precise and repeatable judgments on agent performance.

Skills

Consumer e-commerce fluency

Careful, consistent judgment

Clear, concise written English

Job description

About the job

Mercor connects elite creative and technical talent with leading AI research labs. Headquartered in San Francisco, our investors include Benchmark , General Catalyst , Peter Thiel , Adam D'Angelo , Larry Summers , and Jack Dorsey . Position: Shopify Storefront Agent Evaluator Type: Contract Compensation: $40–$50/hour Location: Remote Commitment: up to 5 hours/week

Role Responsibilities

• Evaluate the Storefront Agent by simulating real shopper interactions, including product discovery, comparisons, and inquiries about sizing, stock, shipping, and returns.
• Grade responses based on Relevance , Accuracy , and Safety to ensure they meet shopper expectations and adhere to store policies.
• Provide concise feedback on any failures to ensure clarity for team members not present during evaluation.
• Work independently to deliver precise and repeatable judgments on agent performance.

Qualifications

Must-Have

• Consumer e-commerce fluency — you shop online regularly and understand effective product recommendations.
• Careful, consistent judgment — apply the rubric uniformly across responses.
• Clear, concise written English — articulate reasons for response failures in a sentence or two.

Preferred

• Hands-on Shopify merchant experience — familiarity with shopper behavior on Shopify stores.
• Experience using AI shopping assistants or chat agents — understand common failure points.
• Prior work evaluating, red‑teaming, or annotating AI model outputs.
• Familiarity with retail operations: catalog and variant structure, inventory, shipping, and returns policy.