Data Operations Lead — RLHF

Best AI Tools Wiki; San Francisco, Northern (CA, KY); Hybrid; USD 180,000 - 240,000; Full time

Benefits offered by this job

Equity in a unicorn AI company

Free meals and snacks

Team offsites

Job summary

Scale AI is seeking a Data Operations Lead to manage large-scale RLHF data annotation programs for frontier AI companies. You will design annotation guidelines, build quality assurance processes, and lead distributed teams of evaluators to produce high-quality training data.

In this role you will collaborate with product and research teams, define metrics for data quality, and scale operations globally while maintaining rigorous data governance and timelines.

Qualifications

• 5+ years of experience in data operations or data quality.
• Experience with human evaluation and annotation workflows.
• Strong project management and team leadership skills.
• Understanding of LLM training data requirements.
• Experience with data quality metrics and frameworks.

Responsibilities

• Design annotation guidelines for RLHF data.
• Build quality assurance processes for data.
• Lead distributed teams of evaluators to produce high-quality training data.

Skills

Data Operations

Quality Assurance

Project Management

Annotation Leadership

Job description

Scale AI is looking for a Data Operations Lead to manage large-scale RLHF data annotation programs for frontier AI companies. You will design annotation guidelines, build quality assurance processes, and lead distributed teams of evaluators to produce high-quality training data.

Requirements

• 5+ years of experience in data operations or data quality
• Experience with human evaluation and annotation workflows
• Strong project management and team leadership skills
• Understanding of LLM training data requirements
• Experience with data quality metrics and frameworks

Nice to Have

• Experience with RLHF or preference data collection
• Background in computational linguistics
• Familiarity with Scale AI's platform
• Experience managing globally distributed teams

Equity in a unicorn AI company

Free meals and snacks

Team offsites

Skills

Data Operations RLHF Quality Assurance Project Management Annotation Leadership