Remote Senior Annotation Ops & Quality Lead

Aquent (United States); New York (NY); Remote; USD 90,000 - 130,000; Full time

Benefits offered by this job

Health benefits

Retirement plan with match

Flexible spending accounts

Job summary

AI Foundation is seeking a Senior Product Quality Analyst / Annotation Operations Specialist to drive high-quality human judgments that ground evaluation and support model fine-tuning. You will lead annotation programs, partner with data science, engineering, and vendors, and build scalable workflows across the team.

The role emphasizes operational ownership, establishment of guidelines, and roadmap development for annotation needs, with a focus on improved evaluation rubrics and collaboration

Qualifications

• Background in data operations (Data Ops), annotation ops, model evaluation, product quality, or technical program management.
• Hands-on experience managing, coordinating, or designing data annotation workflows—this is an operational and program-level ownership role, not a direct labeling position.

Responsibilities

• Lead end-to-end execution of human annotation projects, from intake, scoping, and sourcing to delivery.
• Serve as the primary point of contact across data science, engineering, and partner teams to map annotation requirements to the most efficient internal or vendor pipelines.
• Maintain a roadmap of annotation needs, prioritizing requests and coordinating the appropriate internal or external annotation resources.
• Manage vendors and coordinate contracts, onboarding, and related reviews with legal, security, privacy, finance, and accounting.
• Develop and maintain reusable annotation frameworks and guidelines that can be adopted across teams, and consult on annotation task and rubric design.
• Run feedback sessions with modeling teams to identify model strengths, failure patterns, and evaluation gaps.
• Evaluate annotation quality, identify sources of disagreement, and work with teams and vendors to systematically improve guidelines and outputs.
• Chart best practices and operational documentation for a newly formalized function, partnering closely with dedicated data scientists.

Skills

Data annotation workflows

AI/ML evaluation rubrics

Vendor onboarding

Tools

Claude

Codex

Job description

AI Foundation is seeking a Senior Product Quality Analyst / Annotation Operations Specialist to drive high-quality human judgments that ground evaluation and support model fine-tuning. You will lead annotation programs, partner with data science, engineering, and vendors, and build scalable workflows across the team.

The role emphasizes operational ownership, establishment of guidelines, and roadmap development for annotation needs, with a focus on improved evaluation rubrics and collaboration