Annotation Quality Program Manager

shift; India; On-site; INR 400,000 - 700,000; Full time

Job summary

shift is seeking a senior quality and annotation lead to own the end-to-end quality bar for egocentric training data. You will develop golden sets, SOPs, and calibration systems, collaborating with engineering to deploy new annotation workflows and QC dashboards.

You will oversee annotator training, run calibration sessions, and design human-in-the-loop processes where models pre-label and humans review, correct, and elevate. Strong leadership and cross-functional collaboration are essential.

Qualifications

• 7+ years hands-on video annotation and quality control.
• 4+ years in a lead or QA capacity: calibrating annotators, adjudicating disagreements, owning guidelines others work to
• Built 0→1 annotation programs and taken them from pilot to production
• Experience integrating autolabeling and model-based annotation into human workflows; building human-in-the-loop pipelines to raise throughput without sacrificing quality.
• Strong cross-functional partnership with product, engineering, and research/ML.
• Authored SOPs and rubrics, with deep command of quality methodology: IAA frameworks, gold-set creation, sampling strategies, tolerance thresholds

Responsibilities

• Own the quality bar end-to-end: golden/benchmark sets, acceptance criteria, frame-level tolerance thresholds, and audit loops.
• Author and version annotation SOPs, rubrics, and edge-case rules as the client spec evolves.
• Measure and report quality: IAA (kappa/alpha, tolerance-based agreement on temporal boundaries), acceptance/rejection, rework, cost-per-accepted-hour.
• Set up audit frameworks for quality control and drive human-in-the-loop operations to detect bad quality vs good quality data.
• Stand up annotator training and certification, run calibration sessions, and design human-in-the-loop workflows where models pre-label, and humans review, correct, and elevate.
• Partner with engineering and product on new workflows (hand tracking, object identification, action labeling): label schemas, tool requirements, QC dashboards, and pilot validation before production scaling

Skills

Video annotation

Quality control

QA leadership

SOP authoring

Cross-functional partnership

Model-based annotation

Auditing workflows

Tools

CVAT

Labelbox

Encord

V7

Label Studio

Job description

You'll own the quality bar for shift's egocentric training data: the golden sets, SOPs, and calibration system that every annotator, reviewer and labeling model is measured against. You'll also work with engineering to stand up new annotation workflows in quality control, hand tracking, object identification, and action labelling.

What you'll do

• Own the quality bar end-to-end: golden/benchmark sets, acceptance criteria, frame-level tolerance thresholds, and the audit loops that hold them
• Author and version annotation SOPs, rubrics, and edge-case rules as the client spec evolves
• Measure and report quality: IAA (kappa/alpha, tolerance-based agreement on temporal boundaries), acceptance/rejection, rework, cost-per-accepted-hour — and root-cause disagreements to guideline ambiguity vs. annotator error
• Set up audit frameworks for quality control and drive human-in-the-loop operations to detect bad quality vs good quality data
• Stand up annotator training and certification, run calibration sessions, and design human-in-the-loop workflows where models pre-label, and humans review, correct, and elevate
• Partner with engineering and product on new workflows (hand tracking, object identification, action labeling): label schemas, tool requirements, QC dashboards, and pilot validation before production scaling

What we're looking for

• 7+ years hands-on video annotation and quality control
• 4+ years in a lead or QA capacity: calibrating annotators, adjudicating disagreements, owning guidelines others work to
• Built 0→1 annotation programs and taken them from pilot to production
• Experience in integrating autolabeling and model-based annotation into human workflows; building human-in-the-loop pipelines to raise throughput without sacrificing quality.
• Strong cross-functional partnership with product, engineering, and research/ML.
• Authored (not just followed) SOPs and rubrics, with deep command of quality methodology: IAA frameworks, gold-set creation, sampling strategies, tolerance thresholds
• Hands-on with 2+ professional annotation tools (CVAT, Labelbox, Encord, V7, Label Studio, or equivalent), configuring label schemas and QA workflows — including auditing model-generated labels

Bonus

• Experience with egocentric/first-person video for robotics or embodied AI
• Robotics, mechatronics, or computer-vision engineering background
• Managed distributed annotator workforces or external vendors
• Multi-modal annotation (3D/depth, joint pose, grasp outcome classification)
• Trained or fine-tuned auto-labeling models, or built annotation tooling
• Set up quality control and audit frameworks and scaled the operations

Why this role

• The quality bar is the product: shift's ground truth becomes the benchmark physical-AI models are measured against
• Ground-floor ownership at a funded, fast-growing startup — you build the quality system from scratch

About shift

shift pays everyday workers to record short videos of the tasks they already do (in kitchens, workshops, job sites, bakeries, and warehouses) to help train the next generation of AI and robotics. We’ve paid out over $15M to 25,000+ workers across 15+ countries and are now looking to scale further.