LLM Infrastructure Engineer: Scale & Reliability

United States Digital Space LLC; New York, San Francisco (NY, CA); On-site; USD 180,000 - 225,000; Full time

Job summary

Scale is seeking a Software Engineer for the ML Infrastructure team to design and build platforms for scalable, reliable LLM serving. You will work closely with researchers and engineers to deploy models in production and research environments from New York and beyond.

Scale rewards strong backend design, cloud familiarity, and independent problem solving with equity options and benefits. The base salary range for this full-time role in New York, San Francisco, and Seattle is $180,000—$225,000

Qualifications

• 4+ years of experience building large-scale, high-performance backend systems.
• Strong programming skills in one or more languages (e.g., Python, Go, Rust, C++).
• Experience with LLM serving and routing fundamentals (e.g. rate limiting, token streaming, load balancing, budgets, etc.).
• Experience with LLM capabilities and concepts such as reasoning, tool calling, prompt templates, etc.
• Experience with containers and orchestration tools (e.g., Docker, Kubernetes).
• Familiarity with cloud infrastructure (AWS, GCP) and infrastructure as code (e.g., Terraform).
• Proven ability to solve complex problems and work independently in fast-moving environments.

Responsibilities

• Build and maintain fault-tolerant, high-performance systems for serving LLMs workloads at scale.
• Build an internal platform to empower LLM capability discovery.
• Collaborate with researchers and engineers to integrate and optimize models for production and research use cases.
• Conduct architecture and design reviews to uphold best practices in system design and scalability.
• Develop monitoring and observability solutions to ensure system health and performance.
• Lead projects end-to-end, from requirements gathering to implementation, in a cross-functional environment.

Job description

Scale is seeking a Software Engineer for the ML Infrastructure team to design and build platforms for scalable, reliable LLM serving. You will work closely with researchers and engineers to deploy models in production and research environments from New York and beyond.

Scale rewards strong backend design, cloud familiarity, and independent problem solving with equity options and benefits. The base salary range for this full-time role in New York, San Francisco, and Seattle is $180,000—$225,000