Staff Software Engineer, Foundational Model Serving

Cacheflow; San Francisco (CA); On-site; USD 120,000 - 160,000; Full time

Job summary

A leading tech enterprise in San Francisco is seeking a Staff Engineer to shape their foundation model API product. You will design and build systems that ensure efficient performance on high-throughput, low-latency GPU workloads. The ideal candidate will have experience with operational sensitive systems but does not need prior AI experience. Collaboration with various teams is essential to improve product offerings and architectural decisions.

Qualifications

• No prior ML or AI experience is necessary.
• Strong engineering skills required.
• Ability to work in a collaborative environment.

Responsibilities

• Design and build systems for high-throughput, low-latency inference.
• Influence architectural direction for AI model serving.
• Collaborate across various teams to enhance product experience.

Skills

Experience with high scale operational sensitive systems

Interest in building LLM APIs

Experience in customer facing APIs

Job description

Overview

At Databricks, we are passionate about enabling data teams to solve the world's toughest problems — from making the next mode of transportation a reality to accelerating the development of medical breakthroughs. We do this by building and running the world's best data and AI infrastructure platform so our customers can use deep data insights to improve their business.

Foundation Model Serving is the API Product for hosting and serving frontier AI model inference for open source models like Llama, Qwen, and GPT OSS as well as proprietary models like Claude and OpenAI GPT. For this role, no prior ML or AI experience is necessary. We’re looking for engineers who have owned high scale operational sensitive systems like customer facing APIs, Edge Gateways, ML Inference, or similar services and have an interest in getting deep building LLM APIs and runtimes at scale.

As a Staff Engineer, you’ll play a critical role in shaping both the product experience and core infrastructure. You will design and build systems that enable high-throughput, low-latency inference on GPU workloads with frontier models, influence architectural direction, and collaborate closely across platform, product, infrastructure, and research teams to deliver a world-class foundation model API product.

The impact you will have: