Senior Software Engineer, Distributed Cloud AI & LLM

Google; Town of Montana (WI); On-site; USD 174,000 - 252,000; Full time

Job summary

Google’s Senior Software Engineer role focuses on building and optimizing LLM inference serving within Google Distributed Cloud AI. You will drive cross‑team integration across billing, observability, security, and quota management, while collaborating with teams on LLM frameworks, Kubernetes orchestration, networking, and hardware acceleration.

The role emphasizes large-scale infrastructure, distributed systems expertise, and AI/ML applications, with a strong emphasis on performance and

Qualifications

• Bachelor's degree or equivalent practical experience.
• 5 years of experience with software development in one or more programming languages.
• 3 years of experience testing, maintaining, or launching software products, and 1 year of experience with software design and architecture.
• 3 years of experience with developing large-scale infrastructure, distributed systems or networks, or experience with compute technologies, storage or hardware architecture.
• Experience programming in Go for software development, including AI/ML applications.

Responsibilities

• Lead the technical design, development, and optimization of software components critical for LLM inference serving on GDC.
• Drive horizontal integration across core platform services, including billing, logging, observability, security, and quota management.
• Implement and enhance serving capabilities to support advanced LLM techniques like disaggregated serving, speculative decoding, quantization, and efficient model sharding.
• Collaborate closely with internal teams developing core LLM frameworks, container orchestration (Kubernetes, GKE), networking infrastructure, and hardware acceleration.

Skills

Go programming

Distributed systems

AI/ML applications

Software development

Testing

Education

Bachelor's degree

Master's degree or PhD (CS)

Tools

Kubernetes

GKE

Go tooling

Job description

Google’s Senior Software Engineer role focuses on building and optimizing LLM inference serving within Google Distributed Cloud AI. You will drive cross‑team integration across billing, observability, security, and quota management, while collaborating with teams on LLM frameworks, Kubernetes orchestration, networking, and hardware acceleration.

The role emphasizes large-scale infrastructure, distributed systems expertise, and AI/ML applications, with a strong emphasis on performance and