Shopify Ops AI QA Reviewer – Remote

AuraOne; United States; On-site; USD 68,880 - 96,432; Full time

Job summary

AuraOne is seeking a remote Shopify Expert to evaluate AI outputs across specialist operations workflows. You will review responses against playbooks, grade tone and escalation, and flag policy gaps on a structured rubric.

This contractor role requires direct Shopify operations experience, strong attention to detail, and reliable async availability for at least 10 hours per week. Applicants must be authorized to work in the United States and collaborate effectively in a fast-paced, remote

Qualifications

• Direct working experience in Shopify specialist operations on real teams.
• Ability to apply multi-page rubrics consistently across long batches.
• Clear written reasoning naming the policy or workflow applied.
• Strong attention to detail and ability to flag prompt issues.
• Reliable async availability for at least 10 hours per week.

Responsibilities

• Review AI outputs against Shopify specialist workflows, playbooks, and policy.
• Grade tone, escalation logic, and stakeholder fit on a rubric.
• Flag operational risk, missed escalations, and policy-adherence gaps with severity tags.
• Capture the right next step so the modeling team can train on it.
• Adjudicate disputed playbook calls using firm guidance.
• Maintain reviewer-quality scores in inter-rater calibration cycles.

Skills

Operational review

Policy adherence

Workflow judgment

Stakeholder communication

Shopify specialist operations

Job description

AuraOne is seeking a remote Shopify Expert to evaluate AI outputs across specialist operations workflows. You will review responses against playbooks, grade tone and escalation, and flag policy gaps on a structured rubric.

This contractor role requires direct Shopify operations experience, strong attention to detail, and reliable async availability for at least 10 hours per week. Applicants must be authorized to work in the United States and collaborate effectively in a fast-paced, remote