AI Annotation Program Lead | Quality & Ops

Programmers.io; United States; On-site; USD 90,000 - 120,000; Full time

Job summary

Programmers.io is seeking an Annotation Program Lead to manage our AI quality annotation program. You will oversee daily operations, lead a team of annotators reviewing customer conversations and AI interactions, and develop guidelines, procedures, and quality standards.

You will calibrate quality programs, partner with Data Scientists to maintain gold-standard datasets, monitor accuracy and throughput, report to Legal, Compliance and executives, and plan capacity to handle seasonal volumes

Qualifications

• Experience leading a human annotation program and datasets.
• Strong QA and process improvement abilities.
• Proven ability to coordinate with Legal, Compliance, and exec teams.
• Experience setting up annotation jobs and QA before launch.

Responsibilities

• Lead the day-to-day operation of the AI annotation program.
• Manage a team of annotators reviewing customer conversations and AI interactions.
• Develop annotation guidelines, procedures, and quality standards.
• Establish calibration programs and QA processes for consistency across reviewers.
• Partner with Data Scientists to create and maintain gold-standard datasets.
• Monitor annotation accuracy, throughput, and quality metrics.
• Coordinate reporting and deliverables for Legal, Compliance, Responsible AI and executives.
• Manage annotation workflows and tooling, including setting up annotation jobs and testing UIs before launch.
• Plan capacity to support seasonal increases in review volume.
• Identify process improvements to increase efficiency and annotation quality.

Skills

Program management

Quality assurance

Stakeholder engagement

Data annotation operations

Tools

Annotation tooling

Data processing scripts

Job description

Programmers.io is seeking an Annotation Program Lead to manage our AI quality annotation program. You will oversee daily operations, lead a team of annotators reviewing customer conversations and AI interactions, and develop guidelines, procedures, and quality standards.

You will calibrate quality programs, partner with Data Scientists to maintain gold-standard datasets, monitor accuracy and throughput, report to Legal, Compliance and executives, and plan capacity to handle seasonal volumes