Staff Software Engineer, AI Inference

Syllo; United States; Remote; USD 190,000 - 230,000; Full time

Benefits offered by this job

Health insurance

Equity

Job summary

Syllo is scaling its AI platform and seeks a Staff Software Engineer to lead the production inference stack. You will define architecture, evaluate technologies, and build systems that deploy, optimize, and monitor AI models in production.

In this hands-on role you will work with ML, infrastructure, and product teams to establish a robust, scalable foundation for AI model serving, while mentoring engineers and shaping the long-term AI infrastructure strategy.

Qualifications

• Significant experience designing and operating production AI inference systems.
• Experience building or leading production LLM serving infrastructure.
• Proficiency with modern inference runtimes and frameworks (e.g., vLLM, TensorRT-LLM, Triton).
• Strong background in distributed systems or high-performance platform engineering.
• Experience optimizing inference performance (latency, throughput, memory, cost).
• Experience operating GPU infrastructure in production.
• Proficiency in Python and at least one systems language (Go, Rust, or C++).
• Proven ability to lead technical architecture for complex infra initiatives.

Responsibilities

• Lead the design and development of the production inference platform.
• Define the roadmap for inference infrastructure, model serving, and runtime optimization.
• Build and operate scalable, cost-effective systems for serving large language models in production.
• Evaluate and integrate modern inference technologies and runtimes.
• Optimize latency, throughput, GPU utilization, memory efficiency, and costs.
• Develop systems for model deployment, traffic routing, autoscaling, scheduling, and observability.
• Partner with ML engineers to productionize new models and techniques.
• Establish benchmarking methodologies to evaluate models and hardware.
• Decide when to build internally vs. adopt open-source or commercial solutions.
• Mentor engineers and define AI infra best practices.

Skills

LLM serving

Distributed systems

Python

Systems programming

Model deployment

Performance optimization

Production AI infra

Tools

vLLM

SGLang

TensorRT-LLM

Triton Inference Server

Hugging Face TGI

NVIDIA Dynamo

Job description

About Syllo

Syllo is on a mission to transform litigation. Our product is a unified litigation platform that enables lawyers and paralegals to safely harness the power of language models and agentic AI throughout the litigation life cycle. Since going to market, we have gained a diverse group of enterprise customers, including some of the biggest law firms and corporations in the country, and we are quickly expanding. By reducing the expense of litigation industry-wide, we aim to improve access to high-quality representation and promote the alignment of legal outcomes with merit.

About the Role

As we continue to scale our AI platform, we’re investing in our own inference stack to deliver best-in-class performance, reliability, cost efficiency, and flexibility across the latest generation of open-source language models.

We’re looking for a Staff Software Engineer to spearhead this effort.

You’ll define the architecture, evaluate emerging technologies, and build the systems that power model serving. You’ll partner closely with machine learning, infrastructure, and product engineering to establish the foundation for how AI models are deployed, optimized, monitored, and operated in production.

This is a highly hands-on technical role. You’ll spend the majority of your time designing, building, and optimizing production systems while helping shape our long-term AI infrastructure strategy.

Responsibilities

• Lead the design and development of our production inference platform.
• Define the technical roadmap for inference infrastructure, model serving, and runtime optimization.
• Build and operate scalable, cost-effective systems for serving large language models in production.
• Evaluate and integrate modern inference technologies, frameworks, and serving runtimes.
• Optimize latency, throughput, GPU utilization, memory efficiency, and infrastructure cost.
• Develop systems for model deployment, traffic routing, autoscaling, scheduling, observability, and operational excellence.
• Partner with ML engineers to productionize new models and inference techniques.
• Establish benchmarking methodologies to evaluate new models, runtimes, and hardware.
• Make key architectural decisions around when to build internally versus leverage open-source or commercial solutions.
• Mentor engineers as the team grows and help establish engineering best practices for AI infrastructure.

Qualifications

• Significant experience designing and operating production AI inference systems.
• Experience building or leading production LLM serving infrastructure.
• Deep experience with one or more modern inference runtimes and frameworks such as vLLM, SGLang, TensorRT-LLM, Triton Inference Server, Hugging Face TGI, NVIDIA Dynamo, or comparable technologies.
• Strong background in distributed systems, backend infrastructure, or high-performance platform engineering.
• Experience optimizing inference performance across GPU workloads, including latency, throughput, batching, memory utilization, and serving efficiency.
• Experience operating GPU infrastructure in production.
• Strong proficiency in Python and at least one systems programming language (such as Go, Rust, or C++).
• Proven ability to lead technical architecture for complex infrastructure initiatives.
• Excellent communication skills and the ability to influence technical direction across engineering teams.

Salary Range ($190- $230K) plus health insurance and equity.

United States - Remote Pay Range

$190,000—$230,000 USD