Senior Software Engineer, AI Compute Scheduling

NVIDIA Corporation; Austin (TX); In loco; USD 224.000 - 357.000; Tempo pieno

Descrizione del lavoro

NVIDIA Corporation in Austin, TX is seeking a Senior Software Engineer to design and build scheduling for large GPU fleets using Slurm and Kubernetes, extending batch scheduling for AI training and inference. You'll own parts of the scheduling stack, build policy layers for fair sharing, run Slurm on Kubernetes, and contribute to open-source projects while collaborating with researchers and engineers.

The role emphasizes reliability, on-call responsiveness, and measurable improvements in GPU

Competenze

• BS or MS in CS/CE/EE or related field, or equivalent experience.
• 6+ years building production systems software.
• Deep experience with at least one batch or cluster scheduler (e.g., Slurm, Kueue, Volcano, KAI, YuniKorn).
• Strong programming in Go, C/C++, plus Python for tooling and analysis.
• Hands-on Kubernetes experience: controllers or operators, CRDs, scheduling and resource management.
• Solid Linux systems knowledge: containers (Docker, Podman, Enroot or Apptainer), cgroups, networking basics and debugging under load.
• A track record of owning a service in production and making it measurably better.
• Clear written and spoken communication with engineers and researchers.

Mansioni

• Design and build scheduling for large GPU fleets on Slurm and Kubernetes.
• Build the policy layer: fair share, quotas and allocations across research teams.
• Run Slurm as a cloud-native service on Kubernetes, and extend Kubernetes batch scheduling for training and inference.
• Build services around the scheduler: reservation automation, node-health-aware placement, workflow orchestration, and APIs.
• Measure and raise goodput: GPU occupancy, waste, fairness and time to start for large jobs.
• Own reliability for the components you build: on-call, incident analysis and fixes.
• Work upstream and across teams: contribute to Slurm, Kubernetes and related open-source projects.

Conoscenze

Go

C/C++

Python

Kubernetes

Slurm

Kueue

Linux

Cloud

Formazione

BS/MS in CS/CE/EE or related field

Strumenti

Docker

Podman

Enroot

Apptainer

Kubernetes Operators/CRDs

Slurm/Open-source schedulers

Descrizione del lavoro

Senior Software Engineer, AI Compute Scheduling page is loaded## Senior Software Engineer, AI Compute Schedulinglocations: US, CA, Santa Clara: US, MA, Westford: US, TX, Austin: US, NC, Durhamtime type: Full timeposted on: Posted Todayjob requisition id: JR2029701NVIDIA is a pioneer in accelerated computing, known for inventing the GPU and driving breakthroughs in gaming, computer graphics, high-performance computing, and artificial intelligence. Our technology powers everything from generative AI to autonomous systems, and we continue to shape the future of computing through innovation and collaboration. Within this mission, our team, Managed AI Research Superclusters (MARS), builds and scales the infrastructure, platforms, and tools that enable researchers and engineers to develop the next generation of AI/ML systems. By joining us, you'll help design solutions that power some of the world's most advanced computing workloads.The MARS Scheduling team decides how NVIDIA's AI research compute gets used. That fleet spans many of the largest GPU clusters in the world, across multiple clouds and partners, running everything from frontier-model pretraining to reinforcement learning and agentic workloads. We run Slurm and Kubernetes at large scale. Our scheduling is coordinated and topology-sensitive on rack-scale systems. We manage fair sharing and quotas for hundreds of research teams. We support preemption and reservations. Elastic jobs recover from failure without losing progress. You will own real parts of the scheduling stack, and the opportunity to ship code into our open-source and internal platform code, and see the result in GPU-hours returned to researchers within days. The researchers training NVIDIA's own models are your users, and they will tell you quickly what works.**What you'll be doing:*** Design and build scheduling for large GPU fleets on both Slurm and Kubernetes. That covers queueing, gang and topology-aware placement on rack-scale NVLink systems, preemption, reservations and backfill that reaches the end of very deep queues.* Build the policy layer: fair share, quotas and allocations across many research teams, driven from a central allocation system and enforced the same way on every scheduler.* Run Slurm as a cloud-native service on Kubernetes, and extend Kubernetes batch scheduling (Kueue and its ecosystem) for training, reinforcement learning, sandboxes and inference.* Build services around the scheduler: reservation automation, node-health-aware placement, workflow orchestration across clusters, and APIs that let other platforms submit and track work.* Measure and raise goodput: GPU occupancy, waste, fairness and time to start for large jobs. Use the data to choose what to build next.* Own reliability for the components you build: on-call, incident analysis and fixes that remove whole classes of failure, so problems are found before researchers hit them.* Work upstream and across teams: contribute to Slurm, Kubernetes and related open-source projects, and partner with platform, storage, networking and research teams.**What we need to see:*** BS or MS in Computer Science, Computer Engineering, Electrical Engineering or a related field, or equivalent experience.* 6+ years building production systems software.* Deep experience with at least one batch or cluster scheduler, such as Slurm, Kueue, Volcano, KAI, YuniKorn or a scheduler you built. You should be able to explain its internals, not only its configuration.* Strong programming in Go, C or C++, plus Python for tooling and analysis.* Hands-on Kubernetes experience: controllers or operators, CRDs, scheduling and resource management.* Solid Linux systems knowledge: containers (Docker, Podman, Enroot or Apptainer), cgroups, networking basics and debugging under load.* A track record of owning a service in production and making it measurably better.* Clear written and spoken communication with engineers and researchers.**Ways to stand out from the crowd:*** Experience scheduling large-scale AI training: thousands of GPUs, NVLink and InfiniBand or RoCE topology, multi-node jobs that must start together.* Contributions to Slurm, Kubernetes SIG Scheduling, Kueue, Volcano, KAI or similar open-source schedulers.* Experience running Slurm on Kubernetes (Slinky or similar), or building Kubernetes operators for batch or HPC workloads.* Familiarity with RL, agentic or inference workloads and what they need from a scheduler: heterogeneous jobs, sandboxes, elastic scale and fast restarts.* Experience with workflow orchestrators such as Flyte, Airflow, Ray or Kubeflow at scale.Your base salary will be determined based on your location, experience, and other objective job-related factors. The expected base salary ranges for this position are:184,000 USD - 287,500 USD for Level 4; and 224,000 USD - 356,500 USD for Level 5.You will also be eligible for equity and .Applications for this job will be accepted at least until November 14, 2026.This posting is for an existing vacancy.NVIDIA uses AI tools in its recruiting processes.NVIDIA is committed to fostering an inclusive work environment and proud to be an equal opportunity employer. As we highly value diversity in our current and future employees, we do not discriminate (including in our hiring and promotion practices) on the basis of race, religion, color, national origin, gender, gender expression, sexual orientation, age, marital status, veteran status, disability status or any other characteristic protected by law.