Staff/ Principal Software Engineer (L6/L7): AI Platform Model Serving

Netflix

United States

On-site

USD 180,000 - 270,000

Full time

14 days+

Get a reply from this employer

Your resume and cover letter, tailored to exactly what they’re hiring for.

Get past ATS filters

Job summary

Netflix is rebuilding its AI/ML model serving platform to support real-time, asynchronous, and agentic workloads at global scale. You will lead the architecture and implementation of distributed systems that couple in-house inference with third-party hosting, focusing on latency, throughput and cost efficiency.

You will drive deployment strategies across GPUs, CPUs and multiple model providers, delivering durable, resumable execution for long-running tasks and near-real-time inference.

Qualifications

  • Have 12+ years of experience in AI/ML infrastructure and distributed systems.
  • Ability to guide architectural decisions across multiple teams through data and expertise.
  • Shape long-term technical roadmaps (2–3 years ahead) with a focus on scalability and cost efficiency.
  • Partner with product managers, operations, and executives to balance technical trade-offs with business priorities.
  • Translate dense technical concepts for non-technical stakeholders and produce clear ADRs and proposals.
  • Experience with high-traffic distributed services and infrastructure for AI/ML model inference, with emphasis on availability, security and performance.
  • Experience building scalable model-serving solutions for foundation models powering AI applications; reduce latency and costs.

Responsibilities

  • Lead the technology behind Netflix's GenAI and Foundation Model serving strategy.
  • Architect systems that run at scale and with reliability for real-time and asynchronous workloads.
  • Embed AI capabilities into member-facing products and ensure secure, high-performance execution.
  • Collaborate with teams to optimize routing, decoupled serving, and multi-GPU inference.

Skills

Distributed systems
High-availability
GPU/CPU inference optimization
Kubernetes
Cloud platforms (AWS/Azure/GCP)
Architectural decision making
ADR documentation

Education

BS/MS/PhD in Computer Science, Applied Math, Engineering, or related field

Tools

NVIDIA Triton Inference Server
vLLM
SGLang
Kubernetes
AWS/Azure/GCP
NVIDIA CUDA programming

Job description

At Netflix, our mission is to entertain the world. Together, we are writing the next episode - pushing the boundaries of storytelling, global fandom and making the unimaginable a reality. We are a dream team obsessed with the uncomfortable excitement of discovering what happens when you merge creativity, intuition and cutting-edge technology. Come be a part of what's next.

Netflix is on the journey to rebuild the AI/ML model serving platform and infrastructure that turns trained models into served predictions across real-time, async, and agentic workloads, at a scale few companies operate at. Supporting many production clusters, from classical to foundation models, spanning latencies from milliseconds rankers to LLM calls, and powering everything members see from homepage ranking to search, messaging, and internal AI productivity tools.

We are re-building a new model serving platform from the ground up, designed from first principles to meet industry standards. It unifies our in-house inference infrastructure, third-party model hosting, and external LLM providers under a single serving, routing, and experimentation layer. Teams get one consistent way to deploy, route, and A/B test any model, regardless of where it runs.

Re-engineering how GPU capacity is provisioned and used. During off-peak periods, the same fleet that serves latency-critical production traffic is repurposed to run open-weight models for internal coding and developer productivity use cases.

This role sits at the center of Netflix's GenAI and Foundation Model serving strategy, with real R&D scope on frontier AI infrastructure. You will tackle the efficiency problems that most directly drive cost per inference and capacity: disaggregated serving, intelligent routing across models and providers, speculative decoding, prompt caching, and multi-GPU inference. These are the levers that turn GPU spend into measurable business value.

You will lead the technology behind some of Netflix's largest bets, embedding AI capabilities into member-facing products at global scale. You will also build durable, resumable execution for long-running agentic workloads. Online and asynchronous/ Near-Real-Time inference is projected to grow fast in 2028, so getting this foundation right now is what keeps costs sustainable and reliability high as demand scales.

Looking for deep expertise in distributed systems and high-performance RPC, GPU/CPU inference optimization, control planes, and experimentation infrastructure, plus fluency with modern LLM serving stacks and a track record of architecting systems that hold together under both extreme scale and rapid technology change.

Qualifications and Expectations
  • Have a BS/MS/PhD in Computer Science, Applied Math, Engineering, or a related field. With relevant experience in the AI/ ML Infrastructure and distributed systems for 12+ years.
  • Influence without authority to guide architectural decisions and alignment across multi-teams through trust, data, and demonstrated expertise.
  • Shape long-term technical roadmaps (typically looking 2 to 3 years ahead) and ensure scalability, maintainability, and business alignment.
  • Cross-Functional Partnership: Work closely with product managers, operations, and executive stakeholders to balance technical trade-offs with business priorities.
  • Clear Communication skill to translate dense technical concepts for non-technical stakeholders and produce clear documentation, architecture decision records (ADRs), and proposals.
  • Have experience with high-traffic distributed services and infrastructure for AI/ ML model inference, with a focus on high availability, security, and performance.
  • Have experience building scalable model-serving solutions for Foundation Models that powers generative and predictive AI applications with practical skills in reducing latency and costs.
  • Familiar with deploying AI/ ML models using platform and solutions such as NVIDIA Triton Inference Server, vLLM, or SGLang, and scaling them via Kubernetes.
  • Have experience working with public cloud infrastructure providers such as AWS, Azure, or GCP.
Preferred Qualifications
  • Have open-source contributions to projects in the AI/ML, distributed systems, or infrastructure space.
  • Hands-on experience with building AI models, and have explored optimization techniques for AI models, such as quantization, model compression, or model pruning.
  • Proficient in leveraging AI coding tools in software development.
  • Have experience working with model gateway and intelligent routing for choosing the right AI models based on context of the requests.
  • Exposure to deploying multi-adapters model serving, disaggregated prefill/decode, or large-scale KV cache management.
  • Working with evaluation and benchmarking tools for LLMs and AI Agents.
  • Knowledge of GPU topology, NVIDIA CUDA programming, and machine learning compilers.
Key technologies & ecosystem
  • Deep Learning fram
Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Engineering Manager, Online Serving and Inference, AI Platform
Engineering Manager, Online Serving and Inference, AI Platform

pofft Technology Inc. • United States

On-site
USD 523,000 - 920,000
Principal AI Infrastructure Engineer, GenAI Platform
Principal AI Infrastructure Engineer, GenAI Platform

Netflix • United States

On-site
USD 180,000 - 270,000
Principal AI Product Engineer
Principal AI Product Engineer

nscaleoperationsukltd • Georgia

On-site
USD 260,000 - 360,000
Sr. Staff Software Engineer, AI Infrastructure
Sr. Staff Software Engineer, AI Infrastructure

Linkedin3 • Sunnyvale (CA)

Hybrid
USD 260,000 - 360,000
AI/ML Engineer – (Next-Generation AI Platforms & Workloads)
AI/ML Engineer – (Next-Generation AI Platforms & Workloads)

Veear Projects Inc. • Sunnyvale (CA)

On-site
USD 140,000 - 210,000
Senior Deployment Engineer (LLM Inference & Optimization)
Senior Deployment Engineer (LLM Inference & Optimization)

Coreweave • Sunnyvale (CA)

On-site
USD 180,000 - 240,000
Staff AI Product Engineer
Staff AI Product Engineer

nscaleoperationsukltd • Georgia

On-site
USD 180,000 - 240,000
Senior AI (Infrastructure) Engineer | Prop Trading Firm
Senior AI (Infrastructure) Engineer | Prop Trading Firm

Techfellow Limited • New York (NY)

On-site
USD 360,000 - 440,000
Member of Technical Staff (Software Engineer, GPU Cluster Infrastructure)
Member of Technical Staff (Software Engineer, GPU Cluster Infrastructure)

Perplexity • United States

Remote
USD 180,000 - 320,000
Member of Technical Staff (Software Engineer, GPU Cluster Infrastructure)
Member of Technical Staff (Software Engineer, GPU Cluster Infrastructure)

Perplexity • United States

On-site
USD 180,000 - 240,000