Senior Technical Program Manager, Human-in-the-Loop Operations

Roblox; California (MO); On-site; USD 226,000 - 277,000; Full time

Job summary

Roblox is seeking a Senior Technical Program Manager for HITL Operations to own the end-to-end lifecycle of AI data labeling and model evaluation. You will scale data operations, manage a distributed contractor workforce, and drive platform evolution toward AI-assisted workflows.

You will partner with ML Engineers, Data Scientists, and Product Managers to translate model needs into executable data programs, define ground truth, and monitor quality metrics across the organization.

Qualifications

• 5+ years of experience in Technical Program Management, Data Operations, or Operations Management within an AI/ML or data-intensive environment.
• Hands-on data labeling, annotation, or model evaluation with scalable quality control.
• Proven experience managing external vendor relationships and distributed contractor workforces.
• Strong understanding of the ML lifecycle and role of human-labeled data in training and evaluating models.
• Proficiency in SQL and Python for querying datasets and building operational dashboards.
• Excellent written and verbal communication to present data concepts to researchers and business stakeholders.

Responsibilities

• Lead data programs end-to-end across labeling and model evaluation workflows for Roblox's AI teams.
• Define and maintain ground truth with labeling guidelines and quality frameworks for diverse data types.
• Manage a multi-vendor, distributed contractor workforce and forecast budgets.
• Drive the shift to AI-assisted workflows with LLM-based pre-labeling and evaluation.
• Collaborate with ML Engineers, Data Scientists, and Product Managers to translate needs into executable data programs.
• Define and track SLAs, surface risks, and drive continuous process improvement.
• Shape platform and tooling strategy, including vendor pilots and internal platform improvements.

Skills

TPM experience

Data Operations

SQL

Python

Budget management

Vendor management

Tools

Cursor

Claude

Gemini

Label Studio

Scale AI

Job description

Every day, tens of millions of people come to Roblox to explore, create, play, learn, and connect with friends in 3D immersive digital experiences– all created by our global community of developers and creators.

At Roblox, we’re building the tools and platform that empower our community to bring any experience that they can imagine to life. Our vision is to reimagine the way people come together, from anywhere in the world, and on any device. We’re on a mission to connect a billion people with optimism and civility, and looking for amazing talent to help us get there.

A career at Roblox means you’ll be working to shape the future of human interaction, solving unique technical challenges at scale, and helping to create safer, more civil shared experiences for everyone.

As Senior Technical Program Manager for the Human-in-the-Loop (HITL) Operations, you will own the end-to-end lifecycle of Roblox's AI data labeling and model evaluation ecosystem. This is a high-leverage, high-ownership role at the intersection of Research, Engineering, and Operations. You will scale our data operations that is handling million of items evaluated annually today with projection of 4x the current volume by 2030, managing a multi-million annual budget and a distributed workforce of 100+ remote contractors — all while driving the platform evolution from manual workflows to AI-assisted, LLM-accelerated operations.

This is a rare opportunity to build the data infrastructure that directly determines the quality of Roblox's AI models across creator tools, content discovery, in-experience AI, and 3D generative content — at the exact moment when human judgment is the critical ingredient for getting these models right.

You Will

• Lead data programs end-to-end. Own the full lifecycle of labeling and model evaluation workflows across Roblox's AI teams — from translating ambiguous ML requirements into structured annotation tasks, to overseeing contractor execution, quality review, and dataset delivery to model teams.

• Define and maintain ground truth. Develop and iterate on labeling guidelines, annotation schemas, and quality frameworks tailored to Roblox's unique data types: 3D mesh quality, texture coherence, search relevance, game novelty detection, NPC behavior evaluation, and AI-generated Luau code assessment.

• Manage a multi-vendor, distributed workforce. Oversee remote contractor teams, managing workforce allocation, productivity SLAs, quality calibration, and budget forecasting serving across multiple teams with Roblox.

• Drive the shift to AI-assisted workflows. Partner with Engineering to evaluate and implement LLM-based pre-labeling, LLM-as-a-Judge evaluation, and automated QA routing — reducing FTE coordination overhead and scaling throughput proportional to our annual volume growth.

• Partner cross-functionally across Roblox AI. Work closely with ML Engineers, Data Scientists, and Product Managers to translate model development needs into concrete, executable data programs, and communicate program status and quality trends to senior leadership.

• Own operational excellence and KPIs. Define and track SLAs across throughput, inter-annotator agreement, cost per label, and data quality metrics. Surface risks, resolve bottlenecks, and drive continuous process improvement.

• Shape platform and tooling strategy. Contribute to Roblox's next-generation labeling and evaluation platform strategy — including vendor pilots, internal platform improvements, and the transition to a hybrid human-AI collaborative workflow model.

You Have

• 5+ years of experience in Technical Program Management, Data Operations, or Operations Management within an AI/ML or data-intensive environment.

• Deep hands‑on experience with data labeling, annotation, or model evaluation — including designing annotation schemas, writing labeling guidelines, and managing quality control at scale.

• Proven experience managing external vendor relationships and distributed contractor workforces, including workforce planning, quality oversight, and budget management.

• Strong understanding of the ML lifecycle and the role of human-labeled data in training, fine‑tuning, and evaluating models.

• Proficiency in SQL and Python for querying datasets, analyzing label quality, and building operational dashboards. At a minimum, be able to use AI tools to generate queries.

• Excellent written and verbal communication skills — able to write rigorous technical specifications and present complex data concepts to both ML researchers and business stakeholders.

• Proven ability to drive cross-functional alignment and exercise strong judgment, knowing when to champion collaborative efforts versus leading independently.

<
• Exceptional critical thinking skills with a demonstrated capacity to navigate ambiguity and execute effectively in 0-to-1 environments without structured guidance.

• Solid grasp of software development life cycles and hands‑on experience leveraging AI-assisted tools like Cursor, Claude, or Gemini to accelerate workflows.

Nice to Have

• Experience with LLM-based labeling pipelines, LLM-as-a-Judge evaluation, or human-AI calibration loops.

• Familiarity with labeling platforms such as Label Studio, Scale AI, or Snorkel AI.

• Experience evaluating 3D, games, videos, or multimodal data beyond standard text and image annotation.

• Background in vendor platform evaluation or build vs. buy analysis.

• Experience with data operations at a consumer platform operating at massive scale (100M+ users).

• Knowledge of Roblox platform and its games, or general gaming experience.

For roles that are based at our headquarters in San Mateo, CA: The starting base pay for this position is as shown below. The actual base pay is dependent upon a variety of job-related factors such as professional background, training, work experience, location, business needs and market demand. Therefore, in some circumstances, the actual salary could fall outside of this expected range. This pay range is subject to change and may be modified in the future. All full-time employees are also eligible for equity compensation and for benefits as described on this page.

Annual Salary Range

$226,400—$277,350 USD

Roles that are based in an office are onsite Tuesday, Wednesday, and Thursday, with optional presence on Monday and Friday (unless otherwise noted).

Roblox provides equal employment opportunities to all employees and applicants for employment and prohibits discrimination and harassment of any type without regard to race, color, religion, age, sex, national origin, disability status, genetics, protected veteran status, sexual orientation, gender identity or expression, or any other characteristic protected by federal, state or local laws. Roblox also provides reasonable accommodations to candidates with qualifying disabilities or religious beliefs during the recruiting process.

For US based roles only, please note the Company may not be able to employ candidates for this role who have United States work authorization related to certain U.S. visa categories, or support future H-1B sponsorship at this time.