Senior Software Engineer: LLM Token Quota & Infra

Socket.dev; Sunnyvale (CA); On-site; USD 174,000 - 252,000; Full time

Job summary

Google is seeking software engineers to help design and operate globally distributed systems at massive scale. The role emphasizes building scheduling, token-based orchestration, and fair-sharing for AI agents, with opportunities to influence architecture across Cloud services.

Candidates should have strong experience in software development, distributed systems, testing, and leadership potential, with exposure to Java or C/C++.

Qualifications

• Bachelor’s degree or equivalent practical experience.
• 5 years of experience with software development in one or more programming languages.
• 3 years of experience testing, maintaining, or launching software products, and 1 year of experience with software design and architecture.
• 3 years of experience with developing large-scale infrastructure, distributed systems or networks, or experience with compute technologies, storage or hardware architecture.

Responsibilities

• Design and operate globally distributed systems paths that handle millions of queries across multiple regions.
• Build distributed scheduling, dynamic reservation, and fair-sharing tuned to how LLMs actually behave — token-by-token streaming, bursty agent loops, speculative decoding, and prompt caching.
• Build self-healing systems with solid failover, fast coordination, and the telemetry needed to hit five-nines (99.999%) availability.
• Set the architectural direction for token orchestration and distribution across Cloud.

Skills

Software development

Testing and launching software

Software design and architecture

Distributed systems

Education

Bachelor's degree or equivalent

Master's degree or PhD in CS or related field

Tools

Java

C/C++

Job description

Google is seeking software engineers to help design and operate globally distributed systems at massive scale. The role emphasizes building scheduling, token-based orchestration, and fair-sharing for AI agents, with opportunities to influence architecture across Cloud services.

Candidates should have strong experience in software development, distributed systems, testing, and leadership potential, with exposure to Java or C/C++.