Senior Software Engineer - AI Token & Rate-Limit Infra

Google Inc.; Sunnyvale (CA); On-site; USD 174,000 - 252,000; Full time

Job summary

Google Sunnyvale, CA, USA is seeking a Senior Software Engineer for Rate Limiting and Token Management to design scalable infrastructure and governance for AI workloads.

You will work on LLM token quota management, backend protection, and fair-share scheduling across Cloud services. The role requires 5+ years of software development and 3+ years in distributed systems, with Java/C++ as preferred skills.

Qualifications

• Bachelor’s degree or equivalent practical experience.
• 5+ years of experience with software development in one or more programming languages.
• 3 years of experience testing, maintaining, or launching software products, and 1 year of experience with software design and architecture.
• 3 years of experience with developing large-scale infrastructure, distributed systems or networks, or experience with compute technologies, storage or hardware architecture.

Responsibilities

• Design and operate globally distributed systems paths that handle millions of queries across multiple regions.
• Build distributed scheduling, dynamic reservation, and fair-sharing tuned to how LLMs actually behave — token-by-token streaming, bursty agent loops, speculative decoding, and prompt caching.
• Build self-healing systems with solid failover, fast coordination, and the telemetry needed to hit five-nines (99.999%) availability.
• Set the architectural direction for token orchestration and distribution across Cloud.

Skills

5+ years software development

3+ years testing/maintaining/launching

3+ years distributed systems

Education

Bachelor’s degree or equivalent

Master’s degree or PhD in CS or related field

Tools

Java

C/C++

Generative AI

Job description

Google Sunnyvale, CA, USA is seeking a Senior Software Engineer for Rate Limiting and Token Management to design scalable infrastructure and governance for AI workloads.

You will work on LLM token quota management, backend protection, and fair-share scheduling across Cloud services. The role requires 5+ years of software development and 3+ years in distributed systems, with Java/C++ as preferred skills.