Software Engineer, Model Inference & LLM Deployment

Google LLC; Mountain View, Northern (CA, KY); Hybrid; USD 180,000 - 300,000; Full time

Job summary

Google DeepMind seeks a Software Engineer for Model Inference to advance production-grade ML serving systems. You will collaborate with ML researchers and engineers to deploy large language models and optimize inference on GPUs/TPUs within Google's infra.

You'll work across model hosting, testing, and performance tuning, contributing to scalable, low-latency serving infrastructure and end-to-end deployment pipelines in a mission-driven team.

Qualifications

• Bachelor’s degree or equivalent practical experience.
• 8 years of experience in software development.
• 2 years of experience in deploying and maintaining ML models in a live production environment.
• Experience in profiling, configuring, or executing ML workloads directly on hardware accelerators (e.g., GPU or TPU).
• Experience designing, building, or optimizing model serving infrastructure or inference backends.

Responsibilities

• Collaborate closely with Research teams to understand next generation modeling approaches, ensuring they are designed and implemented with production considerations in mind.
• Work with infrastructure teams to deliver serving infrastructure that is designed for maximum efficiency and performance, addressing bottlenecks in speed, scale, and quality.
• Identify opportunities to automate tasks, eliminate redundancies, build performant tests, and improve the overall velocity of model releases.
• Gain a deep understanding of serving frameworks, pre-processing pipelines, caching mechanisms, and other relevant technologies.
• Leverage roofline analysis, hardware-level profiling, and systems analysis to identify and eliminate performance bottlenecks across ML frameworks, compilers (XLA), custom kernels (Pallas), and serving infrastructure on hardware accelerators (TPUs/GPUs).

Skills

ML deployment

Performance tuning

Collaboration

Software engineering

Education

Bachelor’s degree or equivalent practical experience

8 years of software development experience

2 years deploying ML models in production

Experience with ML workloads on GPUs/TPUs

Model serving/inference backend design

Tools

JAX

PyTorch

CUDA/OpenCL

XLA

Pallas

Job description

Google DeepMind seeks a Software Engineer for Model Inference to advance production-grade ML serving systems. You will collaborate with ML researchers and engineers to deploy large language models and optimize inference on GPUs/TPUs within Google's infra.

You'll work across model hosting, testing, and performance tuning, contributing to scalable, low-latency serving infrastructure and end-to-end deployment pipelines in a mission-driven team.