Senior Software Engineer (Mk8S Control Plane)

Lambda; United States; On-site; USD 180,000 - 250,000; Full time

Benefits offered by this job

Health, dental, vision

401(k) match

Paid holidays and flexible PTO

Parental, medical, and caregiver leave

Job summary

Lambda is seeking a Senior Software Engineer to join the Managed Kubernetes (Mk8s) team. Build the control plane, operators, and automation for a GPU‑accelerated Kubernetes platform powering AI workloads globally.

Work across compute, network, storage, and security to create reliable, scalable orchestration services and production-grade tooling. You will shape architecture and reliability for the Kubernetes-based infrastructure, collaborating with NVIDIA’s ecosystem and internal teams to deliver

Qualifications

• 6+ years of software engineering experience
• Experience building and operating managed Kubernetes services
• Strong knowledge of Kubernetes internal components (controllers, schedulers, operators)
• Experience with GPU scheduling and HPC workloads on Kubernetes
• Experience with cloud infrastructure and automation tooling

Responsibilities

• Design, build, and maintain scalable control plane services and operators for Kubernetes-based platforms
• Develop automation in Go/Python for end-to-end cluster lifecycle management
• Build GPU-aware orchestration systems and support for AI workloads
• Collaborate with internal teams on networking, storage, and security integration
• Participate in on-call rotation and incident response

Skills

Go

Python

Kubernetes

Distributed systems

Linux

Networking

CNI

CRDs

Operators

Prometheus

Grafana

NVIDIA GPU Operator

Tools

Prometheus

Grafana

NVIDIA GPU Operator

Job description

• We are seeking a Senior Software Engineer to join our Managed Kubernetes (Mk8s) team
• You will play a crucial role in shaping the architecture, reliability, and automation of our Kubernetes-based infrastructure, which powers mission-critical workloads across our global platform
• Lambda is building the AI Cloud of the future
• We are seeking a Senior Software Engineer to help our development of our Managed Kubernetes platform
• Think GKE, but purpose-built for AI workloads and running on bare metal
• In this role, you will help build the infrastructure that powers the next generation of AI training and inference at scale
• As a Senior Engineer on our Orchestration team, you will contribute to Lambda's managed orchestration services, including Managed Kubernetes, Managed Slurm on Kubernetes, and higher-level platform services for inference and AIOps
• You'll work at the intersection of distributed systems, GPU-accelerated computing, and Cloud Native infrastructure to build systems that are reliable, performant, and elegantly simple for our customers
• This is not a role for someone who just operates Kubernetes; it's a role for an engineer who understands how compute, network, storage, and security interact, and can build solutions that account for that context - even while focused primarily on the orchestration layer
• You’ll be working closely with NVIDIA's open-source ecosystem, and partnering with internal teams across the stack to deliver a world-class managed platform
• Design, build, and maintain scalable control plane services, operators, and custom Kubernetes controllers; develop automation in Go/Python for end-to-end cluster lifecycle management - provisioning, upgrades, patching, and deletion
• Build GPU-aware orchestration systems, working within the platform architecture to support GPU scheduling and resource allocation
• Partner with the Network team on networking solutions for AI workloads: CNI integration (Cilium, Multus), high-performance fabrics (InfiniBand, RoCE), RDMA, and GPUDirect
• Write resilient systems that handle failure gracefully - timeouts, retries, backoff, and degraded-mode operation - across large-scale distributed environments
• Develop platform services for inference: model serving infrastructure, autoscaling based on inference load, and multi-model deployment patterns
• Build internal tools and CLIs that let ML/AI teams deploy and monitor their own inference services
• Support and debug production issues through on-call rotation

Benefits

• Health, dental vision
• 401k match
• 5 sick days and 12 paid holidays
• Flexible PTO
• Paid parental, medical, and caregiver leave

Solid knowledge of Linux systems, networking, containers, and cloud infrastructureStrong programming skills in Go and Python; ability to collaborate effectively on shared codebasesSolid grasp of distributed systems fundamentals - fault tolerance, graceful degradation, and failure handling in large-scale environmentsDeep understanding of Kubernetes internals: controllers, schedulers, operators, CRDs, CSI, CNI, and the extension patterns that make Kubernetes powerfulExperience operating the control plane and low-level pieces of large-scale Kubernetes clustersExperience with observability at scale: Prometheus, Grafana, distributed tracing, and building actionable alerting systemsTake pride in owning and delivering core components of products and platformsHave 6+ years of experience in software engineering, with a track record of owning significant technical scope within a team (e.g., driving a project from design through production, or acting as a de facto tech lead on a workstream)Past contributions to CNCF projects or Kubernetes SIGs a plusFamiliarity with HPC and traditional job schedulers (Slurm) and Kubernetes-native batch scheduling (KAI, Volcano, Kueue)Familiarity with GPU, InfiniBand, RDMA, or high-performance computing on KubernetesExperience building and operating managed Kubernetes services (GKE, EKS, AKS, or similar) or working on Kubernetes control plane componentsExposure to storage architecture for AI/ML workloadsHands-on experience with NVIDIA's GPU/networking ecosystem: GPU Operator, device plugins, DCGM, MIG, Network Operator, NCCL tuning, or similar