Senior GenAI Inference Architect & Optimizer

Fdetimes; San Francisco (CA); On-site; USD 191,000 - 233,000; Full time

Job summary

Databricks is seeking a staff software engineer to lead the architecture, development, and optimization of the GenAI inference engine powering the Foundation Model API. You will bridge cutting-edge research and production needs to ensure high throughput, low latency, and robust scaling across GPUs and accelerators.

You will own end-to-end inference components, collaborate with researchers to bring new model features into production, and define instrumentation, profiling, and tracing standards to

Qualifications

• BS/MS/PhD in Computer Science, or a related field.
• Strong software engineering background (6+ years or equivalent) in performance-critical systems
• Proven track record of owning complex system components and driving architectural decisions end-to-end
• Deep understanding of ML inference internals: attention, MLPs, recurrent modules, quantization, sparse operations
• Hands-on experience with CUDA, GPU programming, and key libraries (cuBLAS, cuDNN, NCCL, etc.)
• Strong background in distributed systems design, including RPC frameworks, queuing, RPC batching, sharding, memory partitioning
• Demonstrated ability to uncover and solve performance bottlenecks across layers (kernel, memory, networking, scheduler)
• Experience building instrumentation, tracing, and profiling tools for ML models
• Ability to lead through influence - work closely with ML researchers, translate novel model ideas into production systems
• Excellent communication and leadership skills, with a proactive and ownership-driven mindset
• Bonus: published research or open-source contributions in ML systems, inference optimization, or model serving

Responsibilities

• Own and drive the architecture, design, and implementation of the inference engine, and collaborate on model-serving stack optimized for large-scale LLMs inference
• Partner closely with researchers to bring new model architectures or features (sparsity, activation compression, mixture-of-experts) into the engine
• Lead the end-to-end optimization for latency, throughput, memory efficiency, and hardware utilization across GPUs, and accelerators
• Define and guide standards to build and maintain instrumentation, profiling, and tracing tooling to uncover bottlenecks and guide optimizations
• Architect scalable routing, batching, scheduling, memory management, and dynamic loading mechanisms for inference workloads
• Ensure reliability, reproducibility, and fault tolerance in the inference pipelines, including A/B launches, rollback, and model versioning
• Collaborate cross-functionally on Integrating with federated, distributed inference infrastructure – orchestrate across nodes, balance load, handle communication overhead
• Drive cross-team collaboration: with platform engineers, cloud infrastructure, and security/compliance teams
• Represent the team externally through benchmarks, whitepapers, and open-source contributions

Skills

CS degree or higher

6+ years in performance-critical系统

Ownership of complex system components

ML inference internals

CUDA/GPU programming

Distributed systems design

Instrumentation/tracing tools

Leadership through influence

Communication and collaboration

Open-source contributions (bonus)

Education

BS/MS/PhD in Computer Science or related field

Tools

CUDA libraries (cuBLAS/cuDNN/NCCL)

RPC frameworks

Job description

Databricks is seeking a staff software engineer to lead the architecture, development, and optimization of the GenAI inference engine powering the Foundation Model API. You will bridge cutting-edge research and production needs to ensure high throughput, low latency, and robust scaling across GPUs and accelerators.

You will own end-to-end inference components, collaborate with researchers to bring new model features into production, and define instrumentation, profiling, and tracing standards to