Staff AI Inference Systems Engineer

CoreWeave; Sunnyvale (CA); On-site; USD 188,000 - 275,000; Full time

Benefits offered by this job

Medical, dental, and vision insurance

Company-paid Life Insurance

Voluntary supplemental life insurance

Health Savings Account

Tuition Reimbursement

Employee Stock Purchase Program (ESPP)

Job summary

CoreWeave is seeking a Staff Software Engineer (IC5) on the Inference team in Sunnyvale, CA to lead architecture and performance across services. You will optimize latency, throughput, and GPU utilization while guiding cross-team design and reliability at scale.

You’ll work with distributed systems, Kubernetes infrastructure, and inference workloads, driving memory optimization and batching strategies. A strong background in Go, Python, or C++ is expected.

Qualifications

• 8–12+ years of experience building and operating large-scale distributed systems or cloud platforms
• Proven experience leading cross-team technical initiatives impacting multiple services or organizations
• Strong programming skills in Go, Python, or C++
• Deep expertise in Kubernetes at production scale, including orchestration, scheduling, and service design
• Strong understanding of distributed systems, networking, and performance optimization
• Experience designing and operating low-latency, high-throughput systems with strict P95/P99 latency requirements
• Hands-on experience with inference systems, including batching or micro-batching strategies, caching, and memory optimization
• Experience improving system performance using metrics-driven approaches (e.g., latency, throughput, utilization)
• Familiarity with mixed precision (BF16, FP8) and streaming inference workloads

Responsibilities

• Lead architecture, performance, and reliability across multiple services and teams within the Inference platform
• Optimize inference performance (latency, throughput, and GPU utilization) and memory usage
• Drive cross-team design initiatives and influence engineering direction
• Work deeply in distributed systems and Kubernetes-based infrastructure, focusing on scheduling, batching, and memory optimization

Skills

Go

Python

C++

Kubernetes

Distributed systems

Performance optimization

Networking

Inference systems

Latency and throughput

Tools

CUDA

NCCL

RDMA

NUMA

GPU interconnects

vLLM

Triton

TensorRT-LLM

Ray Serve

TorchServe

Job description

CoreWeave is seeking a Staff Software Engineer (IC5) on the Inference team in Sunnyvale, CA to lead architecture and performance across services. You will optimize latency, throughput, and GPU utilization while guiding cross-team design and reliability at scale.

You’ll work with distributed systems, Kubernetes infrastructure, and inference workloads, driving memory optimization and batching strategies. A strong background in Go, Python, or C++ is expected.