AI Data Ops Lead: Queue Management & Quality

Prolific Academic Ltd; United States; Remote; USD 85,000 - 120,000; Full time

Benefits offered by this job

Remote-friendly

Flexible hours

Professional development

Job summary

Prolific is seeking an experienced Operations Lead to own queues, triage, prioritization, SLA delivery, and quality on AI data programs. You will monitor backlogs, troubleshoot bottlenecks, and coach annotators to improve outcomes.

You will design calibration processes, build dashboards to track throughput and defect rates, and coordinate a distributed team across time zones, ensuring clear guidance and reliable delivery.

Qualifications

• Experience running queues and quality on AI data programs.
• 3+ years in high-throughput operations.
• Ability to identify low-effort or AI-generated responses.
• Proven ability to design and run calibration processes from scratch.
• Proficiency with spreadsheets; SQL/BI tools are a plus.
• Experience managing contributor performance at scale.
• Comfortable with ambiguity and fast-changing priorities.

Responsibilities

• Own work queues: intake, triage, prioritization, task assignment, and SLA delivery.
• Monitor task queue and backlog control.
• Identify and troubleshoot bottlenecks.
• Monitor quality at project and individual annotator level.
• Perform QA tasks such as corrections and feedback.
• Manage annotator performance by unassigning underperformers and retaining high-performers.
• Build dashboards to track throughput, disagreement rates, and defect rates.
• Oversee Slack channels to provide support and answer participant questions.
• Train new annotators on project guidelines.
• Conduct webinars and host office hours for participants.

Skills

Queue management

Quality assurance

Calibration processes

Data analytics

Spreadsheets / BI tools

Contributor performance management

Adaptability to ambiguity

Tools

SQL

Looker

Tableau

BI tools

Job description

Prolific is seeking an experienced Operations Lead to own queues, triage, prioritization, SLA delivery, and quality on AI data programs. You will monitor backlogs, troubleshoot bottlenecks, and coach annotators to improve outcomes.

You will design calibration processes, build dashboards to track throughput and defect rates, and coordinate a distributed team across time zones, ensuring clear guidance and reliable delivery.