Product Quality Analyst

Aquent; New York (NY); Remoto; USD 90.000 - 130.000; Tempo pieno

Vantaggi offerti da questo lavoro

Health benefits

Retirement plan with match

Flexible spending accounts

Descrizione del lavoro

AI Foundation is seeking a Senior Product Quality Analyst / Annotation Operations Specialist to drive high-quality human judgments that ground evaluation and support model fine-tuning. You will lead annotation programs, partner with data science, engineering, and vendors, and build scalable workflows across the team.

The role emphasizes operational ownership, establishment of guidelines, and roadmap development for annotation needs, with a focus on improved evaluation rubrics and collaboration

Competenze

• Background in data operations (Data Ops), annotation ops, model evaluation, product quality, or technical program management.
• Hands-on experience managing, coordinating, or designing data annotation workflows—this is an operational and program-level ownership role, not a direct labeling position.

Mansioni

• Lead end-to-end execution of human annotation projects, from intake, scoping, and sourcing to delivery.
• Serve as the primary point of contact across data science, engineering, and partner teams to map annotation requirements to the most efficient internal or vendor pipelines.
• Maintain a roadmap of annotation needs, prioritizing requests and coordinating the appropriate internal or external annotation resources.
• Manage vendors and coordinate contracts, onboarding, and related reviews with legal, security, privacy, finance, and accounting.
• Develop and maintain reusable annotation frameworks and guidelines that can be adopted across teams, and consult on annotation task and rubric design.
• Run feedback sessions with modeling teams to identify model strengths, failure patterns, and evaluation gaps.
• Evaluate annotation quality, identify sources of disagreement, and work with teams and vendors to systematically improve guidelines and outputs.
• Chart best practices and operational documentation for a newly formalized function, partnering closely with dedicated data scientists.

Conoscenze

Data annotation workflows

AI/ML evaluation rubrics

Vendor onboarding

Strumenti

Claude

Codex

Descrizione del lavoro

Note: This role is fully remote but you must be in EST or CST time zone.

AI Foundation develops the core ML and AI systems that power personalized digital experiences across the platform. These systems rely on human judgment to ground their intelligence in the nuances of culture, personal taste, and real-world user habits.

We're looking for a Senior Product Quality Analyst / Annotation Operations Specialist to deliver high-quality human judgments that provide ground truth for evaluation, support model fine-tuning, and guide the alignment of LLM-based evaluators. You'll lead annotation projects, drive improvements in annotation quality, and establish scalable workflows across the AI Foundation team.

What You'll Do

• Lead end-to-end execution of human annotation projects, from intake, scoping, and sourcing to delivery.
• Serve as the primary point of contact across data science, engineering, and partner teams to map annotation requirements to the most efficient internal or vendor pipelines.
• Maintain a roadmap of annotation needs, prioritizing requests and coordinating the appropriate internal or external annotation resources.
• Manage vendors and coordinate contracts, onboarding, and related reviews with legal, security, privacy, finance, and accounting.
• Develop and maintain reusable annotation frameworks and guidelines that can be adopted across teams, and consult on annotation task and rubric design.
• Run feedback sessions with modeling teams to identify model strengths, failure patterns, and evaluation gaps.
• Evaluate annotation quality, identify sources of disagreement, and work with teams and vendors to systematically improve guidelines and outputs.
• Chart best practices and operational documentation for a newly formalized function, partnering closely with dedicated data scientists.

Who You Are

• Background in data operations (Data Ops), annotation ops, model evaluation, product quality, or technical program management.
• Hands-on experience managing, coordinating, or designing data annotation workflows—this is an operational and program-level ownership role, not a direct labeling position.
• Demonstrated experience at a scaled tech company or working directly with established data annotation vendors.
• Familiarity with ML-driven products and human evaluation of AI/LLM-powered features.
• Proven ability to translate complex product and modeling needs into clear, scalable annotation guidelines and evaluation rubrics.
• Skilled at vendor onboarding, procurement coordination, and cross-functional project tracking.
• Exceptional cross-functional communication skills (a core deciding factor), with a demonstrated ability to navigate ambiguity, align technical teams, and coordinate operational stakeholders.
• A self-starter comfortable stepping into a 0-to-1 environment to formalize and streamline processes.

Role & Intake Details

Direct/Indirect Management: No

Top 3 Hard Skills: End-to-end data annotation workflow management, familiarity with AI/ML evaluation rubrics & guideline creation, vendor onboarding and procurement coordination.

Top 3 Soft Skills: High-level cross-functional communication (critical stakeholder liaison), self-direction/navigating ambiguity in a new role, and rigorous project tracking across enterprise teams.

Target Profiles / Previous Titles: Data Operations Manager, Annotation Ops Lead, Data Program Manager, Model Evaluation Specialist, Technical Product Quality Analyst. (Disqualifier: Pure data labelers without process/guideline ownership).

Nice-to-Have Skills: Experience interacting with AI collaboration tools (e.g., Claude, Codex), basic coding abilities, and experience scaling annotation pipelines from early-stage to operational maturity.

Strategic Impact & Influence: The role delivers human judgments to ground core recommendation models in cultural nuance and user taste. It bridges daily data pipelines with long-term AI strategy across the AI Foundation department, freeing up engineering and data science bandwidth by professionalizing the human-in-the-loop lifecycle.

About Skill

Skill connects the best professional, IT, engineering, financial and administrative talent with the world's biggest brands. Our eligible talent get access to benefits such as health benefit contributions, retirement plans with match and flexible spending accounts.

Skill is an equal-opportunity employer. We evaluate qualified applicants without regard to age, race, color, religion, sex, sexual orientation, gender identity, national origin, disability, veteran status, and other legally protected characteristics. We're about creating an inclusive environment-one where different backgrounds, experiences, and perspectives are valued, and everyone can contribute, grow their careers, and thriv

#LI-SJ1