LLM Inference & Serving Systems Engineer

Netpreme; Massachusetts; On-site; USD 190,000 - 240,000; Full time

Benefits offered by this job

Competitive salary and equity grant

Health, dental, vision coverage foryou

401(k) with immediate vesting

Life/Disability/AD&D insurance

Job summary

Netpreme is hiring to own the inference-serving stack for frontier LLMs, tuning vLLM, SGLang, and TensorRT-LLM, and architecting distributed serving with NVIDIA Dynamo. You will work with hardware and systems engineers to move models from research to production.

This onsite role is based in Santa Clara, CA or Boston, MA, with relocation assistance available. You will benchmark, optimize parallelism, batching, KV caches, and routing to meet latency and SLA targets, contributing to

Qualifications

• BS, MS, or PhD in Computer Science, Computer Engineering, or a related field, or equivalent practical experience.
• 2+ years of engineering experience in LLM inference, ML systems, GPU systems, or related infrastructure.
• Hands-on experience deploying and optimizing at least one LLM inference engine (e.g., vLLM, SGLang, or TensorRT-LLM).
• Solid understanding of LLM inference fundamentals: prefill/decode, batching, KV caching, and latency/throughput tradeoffs.
• Strong Python and Linux skills, plus hands-on experience with Kubernetes and containerized deployments.
• Comfortable benchmarking and troubleshooting systems, and communicating clearly in a small engineering team.

Responsibilities

• Deploy and operate models in production: Run large language and multimodal models using inference engines like vLLM, SGLang, and TensorRT-LLM.
• Own the serving infrastructure: Build and maintain distributed inference systems with NVIDIA Dynamo, Kubernetes, Helm, and Ansible, including upgrades, monitoring, troubleshooting, and reliability.
• Optimize performance: Tune parallelism strategies (TP/EP/DP/PP), batching, KV caching, routing, and scheduling to meet latency, throughput, and SLA targets.
• Benchmark and measure: Build reproducible benchmarks for latency (TTFT, TPOT, E2E), throughput, concurrency scaling, and system efficiency.
• Support diverse workloads: Enable tool calling, agentic and coding workflows, RAG, and long-context applications.
• Bring new tech into production: Evaluate new models, frameworks, and optimizations, and partner with hardware, model, and systems engineers to ship improvements.

Skills

LLM inference

ML systems

GPU systems

Python

Linux

Kubernetes

Containerization

Education

BS/MS/PhD in Computer Science or related field

Tools

NVIDIA Dynamo

vLLM

SGLang

TensorRT-LLM

Helm

Ansible

Docker

Kubernetes

Job description

Netpreme is hiring to own the inference-serving stack for frontier LLMs, tuning vLLM, SGLang, and TensorRT-LLM, and architecting distributed serving with NVIDIA Dynamo. You will work with hardware and systems engineers to move models from research to production.

This onsite role is based in Santa Clara, CA or Boston, MA, with relocation assistance available. You will benchmark, optimize parallelism, batching, KV caches, and routing to meet latency and SLA targets, contributing to