Technical Lead Manager, Applied Science & AI Evals - Games GenAI

Netflix

Los Angeles (CA)

On-site

USD 523,000 - 1,050,000

Full time

40 hours ago
Be an early applicant

A complete application in a minute

Tailored resume and cover letter, ready to send.

Get past ATS filters

Benefits offered by this job

Health plans
Stock options
Mental health support
401(k) match
Flexible time off

Job summary

Netflix Games seeks a Tech Lead to steer Applied Science for GenAI products, from data curation and fine-tuning to production deployment. You will define evaluation standards, oversee offline and online benchmarking, and ensure safe, scalable agentic experiences.

You will direct a team of ML engineers, collaborate across data science and engineering, and communicate results to leadership while shaping architecture and strategy for the games portfolio.

Qualifications

  • PhD in ML, DL, CS or related quantitative field.
  • 5+ years in ML/applied science or software engineering with hands-on model work.
  • Experience leading ML engineers or applied scientists and mentoring teams.

Responsibilities

  • Lead ML Applied Science research and execution for GenAI portfolio.
  • Define AI evaluation strategy and offline/online measurement processes.
  • Collaborate with Data Science on online metrics and telemetry.
  • Set and maintain ML execution standards across projects.
  • Design safety and red-teaming protocols for agentic systems.

Skills

GenAI research
ML model development
Team leadership
Experiment design
Communication

Education

PhD in ML / DL / CS

Tools

LoRA/QLoRA
SFT
RLHF

Job description

At Netflix, our mission is to entertain the world. Together, we are writing the next episode - pushing the boundaries of storytelling, global fandom and making the unimaginable a reality. We are a dream team obsessed with the uncomfortable excitement of discovering what happens when you merge creativity, intuition and cutting-edge technology. Come be a part of what’s next.We're looking for a Tech Lead to lead Applied Science execution and AI evaluation strategy across Netflix Games' agentic products. This role drives - machine learning model development, fine-tuning, optimization, and the offline/online evaluation work that delivers safe, responsible agentic tools and experiences. You will set the ML quality bar for the games GenAI portfolio, drive model work from data curation through deployment and lead a team that executes.Key ResponsibilitiesLead ML Applied Science research & execution for Netflix Games' GenAI portfolio: model selection, data curation, fine-tuning (SFT, DPO, LoRA/QLoRA), and optimization (distillation, quantization) from prototype through production deployment.Define Netflix Games' AI evaluation strategy end to end: benchmark what \"good\" means for a given model or agentic product, how it's measured offline before ship and online after ship, and how that bar gets enforced consistently across projects.Define offline evaluation processes: curated golden datasets, adversarial and edge-case test suites, automated scoring pipelines, LLM-as-a-judge frameworks, human-in-the-loop grading rubrics, and regression gates that block quality or safety regressions before they reach players or internal users.Support and collaborate with Data Science team in online evaluation metrics for Agentic systems: production telemetry, A/B and holdout experimentation, user feedback loops, and anomaly and drift detection that catches degradation in live model and agent behavior.Set and maintain ML execution standards across the games GenAI portfolio: model and vendor benchmarking methodology, experimentation rigor, statistical soundness, and documentation practices that other teams can adopt.Apply model development and evaluation discipline across both domains of agentic products: player-facing AI features and internal agentic tools.Design safety and red-teaming evaluation protocols to detect toxicity, hallucination, jailbreaks, and out-of-character or off-policy behavior in both player-facing and internal agentic systems.Communicate model performance, evaluation findings, quality trends, and risk to both technical partners and non-technical stakeholders, including leadership, in a way that drives decisions.QualificationsPhD in Machine Learning, Deep Learning, Computer Science, or a related quantitative field.5+ years of experience in machine learning, applied science, or software engineering, with hands-on depth across both model development (training, fine-tuning, optimization) and evaluation/measurement of ML and GenAI systems.Broad, current technical understanding across model families and modalities - LLMs, vision-language and multimodal models, computer vision foundation models, and reinforcement learning - and the judgment to know which architecture fits a given agentic product problem.Direct, hands-on experience evaluating and benchmarking foundation models against each other on inference latency, zero-shot generalization, accuracy, and deployment cost, to inform build-vs-buy and model-selection decisions.Direct, proven experience with the full LLM fine-tuning lifecycle - SFT, DPO, LoRA/QLoRA, and RLHF for long-horizon tasks - from dataset curation through deployment and ongoing maintenance.Direct, proven experience designing and operating evaluation processes for LLM or agentic systems: offline benchmark suites (including tiered/taxonomy-based evaluation by task complexity), automated scoring pipelines, LLM-as-a-judge frameworks, claim-level hallucination and groundedness verification, and online experimentation (A/B testing, production telemetry, drift detection).Experience with on-device or production inference tradeoffs - latency, throughput, memory/battery footprint, and cost - and how those constraints shape model selection and deployment decisions.Prior experience managing or leading a team of ML engineers or applied scientists - hiring, mentoring, setting technical direction, and being accountable for a team's output.Strong foundation in experimental design, statistics, and causal inference; comfortable defending a measurement methodology under scrutiny.Working understanding of agentic system architectures (multi-step reasoning, tool use, multi-agent orchestration) sufficient to develop and evaluate models for them.Excellent written and verbal communication; able to translate model and evaluation results into a clear point of view for both engineering partners and leadership.Comfortable operating at the intersection of two different problem spaces - consumer-facing game experiences and internal developer/creative tooling - and adapting model and evaluation approach to each.Nice to haveExperience developing or evaluating GenAI systems in a games or interactive entertainment context.Experience combining reinforcement learning with LLM-based planning for autonomous or agentic tasks.Familiarity with Model Context Protocol (MCP) or similar agent-to-system connectivity standards, from a model and evaluation standpoint.Background in responsible AI: safety evaluation, prompt injection defense, content policy compliance.Published research, patents, or public contributions in LLM evaluation, model fine-tuning, multimodal AI, agentic evaluation, or RLHF.Generally, our compensation structure consists solely of an annual salary; we do not have bonuses. You choose each year how much of your compensation you want in salary versus stock options. To determine your personal top of market compensation, we rely on market indicators and consider your specific job family, background, skills, and experience to determine your compensation in the market range. The range for this role is $523,000.00 - $1,050,000.00.Netflix provides comprehensive benefits including Health Plans, Mental Health support, a 401(k) Retirement Plan with employer match, Stock Option Program, Disability Programs, Health Savings and Flexible Spending Accounts, Family-forming benefits, and Life and Serious Injury Benefits. We also offer paid leave of absence programs. Full-time hourly employees accrue 35 days annually for paid time off to be used for vacation, holidays, and sick paid time off. Full-time salaried employees are immediately entitled to flexible time off. See more details about our Benefits here .Netflix is a unique culture and environment. Learn more here .Inclusion is a Netflix value and we strive to host a meaningful interview experience for all candidates. If you want an accommodation/adjustment for a disability or any other reason during the hiring process, please send a request to your recruiting partner.We are an equal-opportunity employer and celebrate diversity, recognizing that diversity builds stronger teams. We approach diversity and inclusion seriously and thoughtfully. We do not discriminate on the basis of race, religion, color, ancestry, national origin, caste, sex, sexual orientation, gender, gender identity or expression, age, disability, medical condition, pregnancy, genetic makeup, marital status, or military service.
Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Member of Technical Staff, Agentic Systems - Games
Member of Technical Staff, Agentic Systems - Games

Netflix • Los Gatos (CA)

On-site
USD 890,000 - 1,690,000
Health plans
Stock options
401(k) matching
+2
Member of Technical Staff, Agentic Systems - Games
Member of Technical Staff, Agentic Systems - Games

Netflix • United States

Remote
USD 600,000 - 1,066,000
Health plans
401(k) match
Stock options
+3
Senior Manager, Strategy and Operations (Games)
Senior Manager, Strategy and Operations (Games)

Netflix • Los Angeles (CA)

On-site
USD 260,000 - 430,000
Manager, Applied AI - Title Launch Management
Manager, Applied AI - Title Launch Management

Netflix • Los Gatos (CA)

On-site
USD 436,000 - 710,000
Health plans
401(k) match
Stock options
+2
Research Engineer 6 - TL, Messaging Algorithms
Research Engineer 6 - TL, Messaging Algorithms

Netflix • Los Gatos (CA)

On-site
USD 180,000 - 280,000
Health plans
401(k) with employer match
Stock option program
+5
Senior Manager, Content & Business Products and Delivery F&S - GenAI Enterprise & Workforce Pro[...]
Senior Manager, Content & Business Products and Delivery F&S - GenAI Enterprise & Workforce Pro[...]

Netflix • Los Gatos (CA)

On-site
USD 350,000 - 560,000
Health plans
Mental health support
401(k) with employer match
+5
Creative Tech Researcher 4
Creative Tech Researcher 4

confidential • United States

On-site
USD 300,000 - 537,000
Health plans
Mental health support
401(k) with employer match
+7
Machine Learning Engineer (L4/L5) - Studio Media Algorithms
Machine Learning Engineer (L4/L5) - Studio Media Algorithms

Netflix • Los Angeles (CA)

On-site
USD 466,000 - 750,000
Health Plans
Mental Health support
401(k) Retirement Plan with employer 2
+5
Senior Portfolio Operations Manager, Games
Senior Portfolio Operations Manager, Games

Netflix • Los Angeles (CA)

On-site
USD 170,000 - 250,000
Health plans
Mental health support
401(k) with match
+4
Senior Research Manager - Innovation & Scale
Senior Research Manager - Innovation & Scale

Netflix • Los Angeles (CA)

On-site
USD 412,000 - 690,000
Health plans
Mental health support
401(k) with employer match
+5