Edge & Hybrid Cloud AI Engineer - LLM Inference Lead

Google; Sunnyvale (CA); On-site; USD 174,000 - 252,000; Full time

Benefits offered by this job

Equity

Benefits

Bonus target

Job summary

Google is hiring software engineers to advance AI at scale, delivering LLM inference serving within the Google Distributed Cloud (GDC) Platform. You will work on critical components, from model lifecycle to efficient data loading and dynamic routing, enabling AI across hybrid cloud environments.

You will collaborate with teams across core platform services, tackle complex systems challenges, and contribute to a scalable serving infrastructure that supports innovative AI workloads.

Qualifications

• Bachelor’s degree or equivalent practical experience.
• 5 years of experience with software development in one or more programming languages.
• 3 years of experience testing, maintaining, or launching software products, and 1 year of experience with software design and architecture.
• 3 years of experience with developing large-scale infrastructure, distributed systems or networks, or experience with compute technologies, storage or hardware architecture.
• Experience programming in Go for software development, including AI/ML applications.

Responsibilities

• Lead the technical design, development, and optimization of software components critical for Large Language Model (LLM) inference serving on GDC. This includes model life-cycle management, efficient data loading, dynamic request routing, and intelligent load balancing.
• Drive horizontal integration across core platform services, including billing, logging, observability, security, and quota management.
• Implement and enhance serving capabilities to support advanced LLM techniques like disaggregated serving, speculative decoding, quantization, and efficient model sharding across distributed hardware.
• Collaborate closely with internal teams developing core LLM frameworks, container orchestration (Kubernetes, Google Kubernetes Engine (GKE)), networking infrastructure, and hardware acceleration to build a cohesive and high-performance serving platform.

Skills

Go programming

AI/ML applications

Distributed systems

Large-scale infrastructure

Education

Bachelor’s degree or equivalent practical experience

Master's degree or PhD in Computer Science or related field

Tools

Kubernetes (K8s)

Google Kubernetes Engine (GKE)

Cloud AI platforms

Job description

Google is hiring software engineers to advance AI at scale, delivering LLM inference serving within the Google Distributed Cloud (GDC) Platform. You will work on critical components, from model lifecycle to efficient data loading and dynamic routing, enabling AI across hybrid cloud environments.

You will collaborate with teams across core platform services, tackle complex systems challenges, and contribute to a scalable serving infrastructure that supports innovative AI workloads.