LLM Serving Platform Engineer

Scaleai; San Francisco, New York (CA, NY); On-site; USD 180,000 - 225,000; Full time

Benefits offered by this job

Health and retirement benefits

Equity/stock options

Learning and development stipend

Generous PTO

Commuter stipend

Health, dental, vision coverage

Job summary

United States Digital Space LLC is seeking a Software Engineer on the ML Infrastructure team to design and build scalable platforms for serving LLMs. You will bridge research and engineering to deliver reliable, high-performance systems that power internal and external use cases across environments.

You will collaborate with researchers and engineers to optimize production deployment, architect fault-tolerant services, and contribute to observability, monitoring, and end-to-end project delivery

Qualifications

• 4+ years of experience building large-scale, high-performance backend systems.
• Strong programming skills in one or more languages (e.g., Python, Go, Rust, C++).
• Experience with LLM serving and routing fundamentals (e.g. rate limiting, token streaming, load balancing, budgets).
• Experience with LLM capabilities and concepts such as reasoning, tool calling, prompt templates, etc.
• Experience with containers and orchestration tools (e.g., Docker, Kubernetes).
• Familiarity with cloud infrastructure (AWS, GCP) and infrastructure as code (e.g., Terraform).
• Proven ability to solve complex problems and work independently in fast-moving environments.

Responsibilities

• Build and maintain fault-tolerant, high-performance systems for serving LLMs workloads at scale.
• Build an internal platform to empower LLM capability discovery.
• Collaborate with researchers and engineers to integrate and optimize models for production and research use cases.
• Conduct architecture and design reviews to uphold best practices in system design and scalability.
• Develop monitoring and observability solutions to ensure system health and performance.
• Lead projects end-to-end, from requirements gathering to implementation, in a cross-functional environment.

Skills

Python

Go

Rust

C++

LLM serving

Load balancing

Rate limiting

Tools

Docker

Kubernetes

AWS

GCP

Terraform

Job description

United States Digital Space LLC is seeking a Software Engineer on the ML Infrastructure team to design and build scalable platforms for serving LLMs. You will bridge research and engineering to deliver reliable, high-performance systems that power internal and external use cases across environments.

You will collaborate with researchers and engineers to optimize production deployment, architect fault-tolerant services, and contribute to observability, monitoring, and end-to-end project delivery