Staff Software Engineer, GenAI Inference Engine

Databricks; United States; Remote; USD 191,000 - 233,000; Full time

Benefits offered by this job

Equity

Job summary

Databricks is seeking a staff software engineer for GenAI inference to lead architecture, development, and optimization of the inference engine powering the Databricks Foundation Model API. You will bridge research advances with production demands to ensure high throughput, low latency, and robust scaling.

You will own the end-to-end inference stack, including kernels, runtimes, orchestration, memory, and integration with frameworks and orchestration systems, while collaborating with researchers

Qualifications

• BS/MS/PhD in CS or related field.
• 6+ years in performance-critical systems.
• Owns complex components and drives decisions end-to-end.
• Deep ML inference internals: attention, MLPs, quantization, sparsity.
• CUDA/GPU programming with cuBLAS/cuDNN/NCCL.
• Distributed systems design: RPC, batching, sharding.
• Identify bottlenecks across kernel, memory, network.
• Instrumentation and profiling tool experience.
• Lead through influence with ML researchers.
• Strong communication and ownership mindset.
• Open-source contributions a plus.

Responsibilities

• Own architecture and design of the inference engine for large-scale LLMs.
• Collaborate with researchers to bring new model features into the engine.
• Lead end-to-end optimization for latency, throughput, memory usage, and hardware utilization.
• Define instrumentation, profiling, and tracing tooling to uncover bottlenecks.
• Architect scalable routing, batching, memory management, and dynamic loading.
• Ensure reliability, reproducibility, and model versioning in pipelines.
• Coordinate across federated/distributed infrastructure and security teams.
• Represent the team through benchmarks, whitepapers, and open-source work.

Skills

Strong software engineering

CUDA GPU programming

Distributed systems design

Instrumentation & profiling

Leadership & collaboration

ML inference understanding

Education

BS/MS/PhD in CS or related

Tools

cuBLAS

cuDNN

NCCL

RPC frameworks

Job description

Databricks is seeking a staff software engineer for GenAI inference to lead architecture, development, and optimization of the inference engine powering the Databricks Foundation Model API. You will bridge research advances with production demands to ensure high throughput, low latency, and robust scaling.

You will own the end-to-end inference stack, including kernels, runtimes, orchestration, memory, and integration with frameworks and orchestration systems, while collaborating with researchers