AI Data Ops Lead: Queue, Quality & Throughput

Prolific Academic Ltd; United States; Remote; USD 90,000 - 130,000; Full time

Job summary

Prolific Academic Ltd is seeking an Operations Lead to oversee high-volume AI data labeling programs, owning queues from intake to SLA delivery and ensuring smooth throughput across distributed annotator teams. You will monitor backlogs, identify bottlenecks, calibrate quality processes, and build dashboards to track throughput, disagreement rates, and defect rates while supporting participants.

This role requires hands-on experience with QA, performance management at scale, and comfort with

Qualifications

• Proven, direct experience running queues and quality on AI data programs.
• 3+ years in high-throughput operations (data labeling, trust & safety, BPO, content ops, or CX ops).
• Ability to identify low-effort or AI-generated responses.
• Proven ability to design and run calibration processes from scratch.
• Proficiency in spreadsheets and data tools (SQL, BI, Looker, Tableau).
• Experience managing contributor performance at scale.
• Comfortable with ambiguity and fast-changing priorities.

Responsibilities

• Own work queues: intake, triage, prioritization, task assignment, and SLA delivery
• Monitor task queue and backlog control
• Identify and troubleshoot bottlenecks
• Monitor quality at project and individual annotator level
• Perform QA tasks such as making corrections and providing feedback
• Manage annotator performance by unassigning underperformers and retaining high-performers
• Build and maintain dashboards to track throughput, disagreement rates, and defect rates
• Oversee Slack channels to provide support and answer participant questions
• Train new annotators on project guidelines
• Conduct webinars and host office hours for participants

Skills

Queue management

AI data programs

Calibration processes

Contributor performance

Ambiguity tolerance

High-throughput operations

Tools

SQL

BI

Looker

Tableau

Job description

Prolific Academic Ltd is seeking an Operations Lead to oversee high-volume AI data labeling programs, owning queues from intake to SLA delivery and ensuring smooth throughput across distributed annotator teams. You will monitor backlogs, identify bottlenecks, calibrate quality processes, and build dashboards to track throughput, disagreement rates, and defect rates while supporting participants.

This role requires hands-on experience with QA, performance management at scale, and comfort with