Staff Software Engineer: GenAI Inference & Scale

Databricks; California (MO); On-site; USD 191,000 - 233,000; Full time

Job summary

Databricks is seeking a Staff Software Engineer to lead the GenAI inference stack for their Foundation Model API. You will own the architecture, design, and optimization of the inference engine, targeting high throughput, low latency, and scalable performance across GPUs and accelerators.

You will collaborate with researchers to deploy new model architectures, drive optimizations across kernels, runtimes, and orchestration, and ensure reliability with instrumentation and monitoring.

Qualifications

• BS/MS/PhD in CS or related field.
• 6+ years in performance-critical systems.
• Experience owning complex components and driving architecture end-to-end.
• Deep ML inference internals: attention, MLPs, etc.
• Hands-on CUDA/GPU programming with libraries (cuBLAS/cuDNN/NCCL).
• Distributed systems design including RPC, queues, batching, sharding, memory partitioning.
• Experience building instrumentation, tracing, profiling tools for ML.
• Ability to lead through influence with researchers and production teams.

Responsibilities

• Own architecture, design, and implementation of the inference engine for large-scale LLMs.
• Partner with researchers to bring new model architectures into the engine.
• Lead end-to-end optimization for latency, throughput, memory, and hardware use.
• Define standards for instrumentation, profiling, and tracing tooling.
• Architect scalable routing, batching, scheduling, and memory management for inference workloads.
• Ensure reliability, reproducibility, and fault tolerance in inference pipelines.

Skills

Performance-critical systems

CUDA programming

Distributed systems design

Instrumentation & profiling

Leadership & collaboration

Education

BS/MS/PhD in Computer Science or related field

Tools

CUDA

cuBLAS

cuDNN

NCCL

Job description

Databricks is seeking a Staff Software Engineer to lead the GenAI inference stack for their Foundation Model API. You will own the architecture, design, and optimization of the inference engine, targeting high throughput, low latency, and scalable performance across GPUs and accelerators.

You will collaborate with researchers to deploy new model architectures, drive optimizations across kernels, runtimes, and orchestration, and ensure reliability with instrumentation and monitoring.