LLM Serving Platform Engineer

scaleai; Greater London; On-site; GBP 90,000 - 130,000; Full time

Job summary

Scale AI is seeking a Software Engineer on the ML Infrastructure team to design and build platforms for scalable, reliable serving of LLMs. You will collaborate with researchers and engineers to productionize models, ensuring high-performance, fault-tolerant systems across environments.

The ideal candidate has strong ML fundamentals and deep backend system design experience, with hands-on skills in Python, Go, Rust, or C++, and experience with Docker, Kubernetes, and cloud infrastructure.

Qualifications

• Strong ML fundamentals and backend system design expertise.
• Experience with LLM serving and routing fundamentals (rate limiting, token streaming, load balancing).
• Experience with containers and orchestration (Docker, Kubernetes).
• Familiarity with cloud infrastructure (AWS, GCP) and infrastructure as code (Terraform).

Responsibilities

• Build and maintain fault-tolerant, high-performance systems for serving LLMs at scale.
• Build an internal platform to empower LLM capability discovery.
• Collaborate with researchers and engineers to productionize models.
• Conduct architecture and design reviews to uphold best practices.
• Develop monitoring and observability solutions for system health and performance.
• Lead projects end-to-end in a cross-functional environment.

Skills

Backend system design

ML fundamentals

Python

Go

Rust

C++

LLM serving

load balancing

token streaming

budgeting

Tools

Docker

Kubernetes

Terraform

AWS

GCP

Job description

Scale AI is seeking a Software Engineer on the ML Infrastructure team to design and build platforms for scalable, reliable serving of LLMs. You will collaborate with researchers and engineers to productionize models, ensuring high-performance, fault-tolerant systems across environments.

The ideal candidate has strong ML fundamentals and deep backend system design experience, with hands-on skills in Python, Go, Rust, or C++, and experience with Docker, Kubernetes, and cloud infrastructure.