Staff AI Inference Platform Engineer (Remote)

Syllo; Oregon; Hybrid; USD 190,000 - 230,000; Full time

Benefits offered by this job

Health insurance

Equity

Job summary

Syllo in the United States is seeking a Staff Software Engineer to lead the design and implementation of our production inference platform. You will shape architecture, evaluate new technologies, and build scalable systems powering model serving across teams.

You will collaborate with ML, infrastructure, and product engineering to deploy, monitor, and optimize AI models in production, ensuring latency, cost efficiency, and reliability as we scale our open-source and enterprise offerings.

Qualifications

• Significant experience designing and operating production AI inference systems.
• Experience building or leading production LLM serving infrastructure.
• Deep experience with one or more modern inference runtimes and frameworks such as vLLM, SGLang, TensorRT-LLM, Triton Inference Server, Hugging Face TGI, NVIDIA Dynamo, or comparable technologies.
• Strong background in distributed systems, backend infrastructure, or high-performance platform engineering.
• Experience optimizing inference performance across GPU workloads, including latency, throughput, batching, memory utilization, and serving efficiency.
• Experience operating GPU infrastructure in production.
• Strong proficiency in Python and at least one systems programming language (such as Go, Rust, or C++).
• Proven ability to lead technical architecture for complex infrastructure initiatives.
• Excellent communication skills and the ability to influence technical direction across engineering teams.

Responsibilities

• Lead the design and development of our production inference platform.
• Define the technical roadmap for inference infrastructure, model serving, and runtime optimization.
• Build and operate scalable, cost-effective systems for serving large language models in production.
• Evaluate and integrate modern inference technologies, frameworks, and serving runtimes.
• Optimize latency, throughput, GPU utilization, memory efficiency, and infrastructure cost.
• Develop systems for model deployment, traffic routing, autoscaling, scheduling, observability, and operational excellence.
• Partner with ML engineers to productionize new models and inference techniques.
• Establish benchmarking methodologies to evaluate new models, runtimes, and hardware.
• Make key architectural decisions around internal build vs. external solutions.
• Mentor engineers as the team grows and establish engineering best practices for AI infrastructure.

Skills

AI inference

LLM serving

Distributed systems

Python

Go/Rust/C++

GPU optimization

Production infrastructure

Technical leadership

Communication

Tools

vLLM

TensorRT-LLM

Triton Inference Server

Hugging Face TGI

NVIDIA Dynamo

Job description

Syllo in the United States is seeking a Staff Software Engineer to lead the design and implementation of our production inference platform. You will shape architecture, evaluate new technologies, and build scalable systems powering model serving across teams.

You will collaborate with ML, infrastructure, and product engineering to deploy, monitor, and optimize AI models in production, ensuring latency, cost efficiency, and reliability as we scale our open-source and enterprise offerings.