Technical Lead Manager, Applied Science & AI Evals - Games GenAI

Netflix

Los Gatos (CA)

On-site

USD 523,000 - 1,050,000

Full time

19 hours ago
Be an early applicant

Get a reply from this employer

Your resume and cover letter, tailored to exactly what they’re hiring for.

Get past ATS filters

Benefits offered by this job

Health Plans
401(k) Retirement Plan
Stock Option Program
Flexible Paid Time Off

Job summary

Netflix is seeking a Member of Technical Staff to lead Applied Science execution and AI evaluation for Games GenAI products. You will drive model development, fine-tuning, optimization, and evaluation from data curation to deployment, setting the ML quality bar across the portfolio.

You will guide safety, evaluation frameworks, and collaboration with Data Science. Strong leadership and communication are essential for cross-team impact.

Qualifications

  • PhD in ML, DL, CS or related quantitative field.
  • 5+ years in ML/applied science or software engineering with hands-on model development and evaluation.
  • Broad knowledge across LLMs, vision-language, multimodal models, CV foundation models, and RL.
  • Direct experience evaluating/b benchmarking foundation models for latency, accuracy, and cost.
  • Experience with full LLM fine-tuning lifecycle: SFT, DPO, LoRA/QLoRA, RLHF.
  • Experience designing evaluation processes for LLM/agentic systems: offline benchmarks, automated scoring, A/B testing.

Responsibilities

  • Lead MLApplied Science research for Netflix Games' GenAI portfolio: model selection, data curation, fine-tuning, and optimization.
  • Define AI evaluation strategy end-to-end: offline and online, ensure consistency across projects.
  • Design offline evaluation: datasets, test suites, automated scoring, LLM-as-a-judge, rubrics, regression gates.
  • Collaborate with Data Science on online metrics: telemetry, experiments, user feedback, drift detection.
  • Set ML execution standards: benchmarking method, rigor, documentation for teams to follow.
  • Oversee both consumer-facing and internal agentic AI systems; ensure safe, responsible deployment.
  • Develop safety and red-teaming protocols for toxicity, hallucination, jailbreaks, off-policy behavior.

Skills

Machine Learning
Applied Science
Model Evaluation
Experiment Design
Statistics
Communication
Leadership
Production Deployment

Education

PhD in ML/CS

Tools

SFT
LoRA/QLoRA
RLHF
A/B Testing

Job description

At Netflix, our mission is to entertain the world. Together, we are writing the next episode - pushing the boundaries of storytelling, global fandom and making the unimaginable a reality. We are a dream team obsessed with the uncomfortable excitement of discovering what happens when you merge creativity, intuition and cutting-edge technology. Come be a part of what’s next.

We're looking for a Member of Technical Staff to lead Applied Science execution and AI evaluation strategy across Netflix Games' agentic products. This role drives - machine learning model development, fine-tuning, optimization, and the offline/online evaluation work that delivers safe, responsible agentic tools and experiences. You will set the ML quality bar for the games GenAI portfolio, drive model work from data curation through deployment and lead a team that executes.

Key Responsibilities
  • Lead ML Applied Science research & execution for Netflix Games' GenAI portfolio: model selection, data curation, fine-tuning (SFT, DPO, LoRA/QLoRA), and optimization (distillation, quantization) from prototype through production deployment.
  • Define Netflix Games' AI evaluation strategy end to end: benchmark what "good" means for a given model or agentic product, how it's measured offline before ship and online after ship, and how that bar gets enforced consistently across projects.
  • Define offline evaluation processes: curated golden datasets, adversarial and edge-case test suites, automated scoring pipelines, LLM-as-a-judge frameworks, human-in-the-loop grading rubrics, and regression gates that block quality or safety regressions before they reach players or internal users.
  • Support and collaborate with Data Science team in online evaluation metrics for Agentic systems: production telemetry, A/B and holdout experimentation, user feedback loops, and anomaly and drift detection that catches degradation in live model and agent behavior.
  • Set and maintain ML execution standards across the games GenAI portfolio: model and vendor benchmarking methodology, experimentation rigor, statistical soundness, and documentation practices that other teams can adopt.
  • Apply model development and evaluation discipline across both domains of agentic products: player-facing AI features and internal agentic tools.
  • Design safety and red-teaming evaluation protocols to detect toxicity, hallucination, jailbreaks, and out-of-character or off-policy behavior in both player-facing and internal agentic systems.
  • Communicate model performance, evaluation findings, quality trends, and risk to both technical partners and non-technical stakeholders, including leadership, in a way that drives decisions.
Qualifications
  • PhD in Machine Learning, Deep Learning, Computer Science, or a related quantitative field.
  • 5+ years of experience in machine learning, applied science, or software engineering, with hands-on depth across both model development (training, fine-tuning, optimization) and evaluation/measurement of ML and GenAI systems.
  • Broad, current technical understanding across model families and modalities - LLMs, vision-language and multimodal models, computer vision foundation models, and reinforcement learning - and the judgment to know which architecture fits a given agentic product problem.
  • Direct, hands-on experience evaluating and benchmarking foundation models against each other on inference latency, zero-shot generalization, accuracy, and deployment cost, to inform build-vs-buy and model-selection decisions.
  • Direct, proven experience with the full LLM fine-tuning lifecycle - SFT, DPO, LoRA/QLoRA, and RLHF for long-horizon tasks - from dataset curation through deployment and ongoing maintenance.
  • Direct, proven experience designing and operating evaluation processes for LLM or agentic systems: offline benchmark suites (including tiered/taxonomy-based evaluation by task complexity), automated scoring pipelines, LLM-as-a-judge frameworks, claim-level hallucination and groundedness verification, and online experimentation (A/B testing, production telemetry, drift detection).
  • Experience with on-device or production inference tradeoffs - latency, throughput, memory/battery footprint, and cost - and how those constraints shape model selection and deployment decisions.
  • Prior experience managing or leading a team of ML engineers or applied scientists - hiring, mentoring, setting technical direction, and being accountable for a team's output.
  • Strong foundation in experimental design, statistics, and causal inference; comfortable defending a measurement methodology under scrutiny.
  • Working understanding of agentic system architectures (multi-step reasoning, tool use, multi-agent orchestration) sufficient to develop and evaluate models for them.
  • Excellent written and verbal communication; able to translate model and evaluation results into a clear point of view for both engineering partners and leadership.
  • Comfortable operating at the intersection of two different problem spaces - consumer-facing game experiences and internal developer/creative tooling - and adapting model and evaluation approach to each.
Nice to have
  • Experience developing or evaluating GenAI systems in a games or interactive entertainment context.
  • Experience combining reinforcement learning with LLM-based planning for autonomous or agentic tasks.
  • Familiarity with Model Context Protocol (MCP) or similar agent-to-system connectivity standards, from a model and evaluation standpoint.
  • Background in responsible AI: safety evaluation, prompt injection defense, content policy compliance.
  • Published research, patents, or public contributions in LLM evaluation, model fine-tuning, multimodal AI, agentic evaluation, or RLHF.

Generally, our compensation structure consists solely of an annual salary; we do not have bonuses. You choose each year how much of your compensation you want in salary versus stock options. To determine your personal top of market compensation, we rely on market indicators and consider your specific job family, background, skills, and experience to determine your compensation in the market range. The range for this role is $523,000.00 - $1,050,000.00.

Netflix provides comprehensive benefits including Health Plans, Mental Health support, a 401(k) Retirement Plan with employer match, Stock Option Program, Disability Programs, Health Savings and Flexible Spending Accounts, Family-forming benefits, and Life and Serious Injury Benefits. We also offer paid leave of absence programs. Full-time hourly employees accrue 35 days annually for paid time off to be used for vacation, holidays, and sick paid time off. Full-time salaried employees are immediately entitled to flexible time off. See more details about our Benefits here.

Netflix is a unique culture and environment. Learn more here.

Inclusion is a Netflix value and we strive to host a meaningful interview experience for all candidates.

If you want an accommodation/adjustment for a disability or any other reason during the hiring process, please send a request to your recruiting partner.

We are an equal-opportunity employer and celebrate diversity, recognizing that diversity builds stronger teams. We approach diversity and inclusion seriously and thoughtfully. We do not discriminate on the basis of race, religion, color, ancestry, national origin, caste, sex, sexual orientation, gender, gender identity or expression, age, disability, medical condition, pregnancy, genetic makeup, marital status, or military service.

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Technical Lead Manager, Applied Science & AI Evals - Games GenAI
Technical Lead Manager, Applied Science & AI Evals - Games GenAI

Netflix • Los Angeles (CA)

On-site
USD 523,000 - 1,050,000
Health Plans
Mental Health support
401(k) Retirement Plan with employer-
+6
Technical Lead, Agentic Systems - Games
Technical Lead, Agentic Systems - Games

Netflix • Los Gatos (CA)

Remote
USD 650,000 - 1,100,000
Health plans
Mental health support
401(k) with employer match
+3
Member of Technical Staff, Agentic Systems - Games
Member of Technical Staff, Agentic Systems - Games

Netflix • Los Gatos (CA)

On-site
USD 890,000 - 1,690,000
Health plans
Stock options
401(k) matching
+2
Member of Technical Staff, Agentic Systems - Games
Member of Technical Staff, Agentic Systems - Games

Netflix • United States

Remote
USD 600,000 - 1,066,000
Health plans
401(k) match
Stock options
+3
Research Engineer 6 - TL, Messaging Algorithms
Research Engineer 6 - TL, Messaging Algorithms

Netflix • Los Gatos (CA)

On-site
USD 180,000 - 280,000
Health plans
401(k) with employer match
Stock option program
+5
Senior Manager, Content & Business Products and Delivery F&S - GenAI Enterprise & Workforce Pro[...]
Senior Manager, Content & Business Products and Delivery F&S - GenAI Enterprise & Workforce Pro[...]

Netflix • Los Gatos (CA)

On-site
USD 350,000 - 560,000
Health plans
Mental health support
401(k) with employer match
+5
Manager, Applied AI - Title Launch Management
Manager, Applied AI - Title Launch Management

Netflix • Los Gatos (CA)

On-site
USD 436,000 - 710,000
Health plans
401(k) match
Stock options
+2
Senior Portfolio Operations Manager, Games
Senior Portfolio Operations Manager, Games

Netflix • Los Angeles (CA)

On-site
USD 170,000 - 250,000
Health plans
Mental health support
401(k) with match
+4
Senior Manager, Strategy and Operations (Games)
Senior Manager, Strategy and Operations (Games)

Netflix • Los Angeles (CA)

On-site
USD 260,000 - 430,000
Creative Tech Researcher 4
Creative Tech Researcher 4

confidential • United States

On-site
USD 300,000 - 537,000
Health plans
Mental health support
401(k) with employer match
+7