AI Infrastructure Engineer

Netpreme; Santa Clara, Boston (CA, MA); On-site; USD 170,000 - 210,000; Full time

Benefits offered by this job

Competitive salary with bonus

Health, Dental, and Vision for you and

401(k) match with immediate vesting

Life, Disability, and AD&D insurance

PTO: 20 days + 15 holidays

Lunch stipend

Office in Santa Clara, CA and Boston,

Relocation assistance

Visa sponsorship

Collaborative learning environment

Job summary

Netpreme is seeking a skilled engineer to own the inference-serving stack for frontier LLM workloads. You will optimize and operate distributed inference pipelines, working with vLLM, SGLang, and TensorRT-LLM, while coordinating with hardware and systems teams.

You’ll build benchmarks, tune performance, and drive production readiness with Kubernetes, NVIDIA Dynamo, Helm, and Ansible, in a fast-moving startup setting.

Qualifications

• BS, MS, or PhD in Computer Science, Computer Engineering, or a related field, or equivalent practical experience.
• 2+ years of engineering experience in LLM inference, ML systems, GPU systems, or related infrastructure.
• Hands-on experience deploying and optimizing at least one LLM inference engine (e.g., vLLM, SGLang, or TensorRT-LLM).
• Solid understanding of LLM inference fundamentals: prefill/decode, batching, KV caching, and latency/throughput tradeoffs.
• Strong Python and Linux skills, plus hands-on experience with Kubernetes and containerized deployments.
• Comfortable benchmarking and troubleshooting systems, and communicating clearly in a small engineering team.

Responsibilities

• Deploy and operate models in production: Run large language and multimodal models using inference engines like vLLM, SGLang, and TensorRT-LLM.
• Own the serving infrastructure: Build and maintain distributed inference systems with NVIDIA Dynamo, Kubernetes, Helm, and Ansible, including upgrades, monitoring, troubleshooting, and reliability.
• Optimize performance: Tune parallelism strategies (TP/EP/DP/PP), batching, KV caching, routing, and scheduling to meet latency, throughput, and SLA targets.
• Benchmark and measure: Build reproducible benchmarks for latency (TTFT, TPOT, E2E), throughput, concurrency scaling, and system efficiency.
• Support diverse workloads: Enable tool calling, agentic and coding workflows, RAG, and long-context applications.
• Bring new tech into production: Evaluate new models, frameworks, and optimizations, and partner with hardware, model, and systems engineers to ship improvements.

Skills

Python

Linux

Education

BS/MS/PhD in CS/CE

Tools

vLLM

SGLang

TensorRT-LLM

Kubernetes

Helm

Ansible

NVIDIA Dynamo

Job description

About Netpreme

The biggest bottleneck in AI inference today is memory. Chips can compute faster than they can be fed data, a problem known as the memory wall, and Netpreme is building the solution. Our approach spans the full stack, from a proprietary memory management library down to X-Mem™, our Memory Processing Unit (MPU) silicon. Netpreme was co-founded by researchers in optical networking and ML systems. We’re a small team in Cambridge, MA and Santa Clara, CA, and we’re looking for people who want to own hard problems end to end and help shape where the company goes next.

About the Role

Serving frontier LLMs efficiently at scale is a moving target: every new model and workload reshuffles the tradeoffs across parallelism, batching, KV caching, and scheduling. You’ll own Netpreme’s inference-serving stack end to end, from tuning vLLM, SGLang, and TensorRT-LLM to architecting distributed serving with NVIDIA Dynamo (prefill/decode disaggregation, KV-cache-aware routing). Working directly with our hardware and systems engineers, you’ll build the benchmarks and drive the optimizations that take new models from research to production.

This role will be performed onsite from one of our offices in Santa Clara, CA or Boston, MA (relocation assistance available).

The Opportunity

• Impact: AI is advancing faster than the infrastructure underneath it. We’re working at that layer, unlocking more capability from every GPU while dramatically improving efficiency. The work you do here compounds across state-of-the-art models, systems, and real-world applications.

• Timing: Joining now means real ownership, including equity, and a direct say in product direction. You’ll work from first principles, move quickly from insight to execution, and see your work show up in what we ship, not just in a roadmap.

• Culture: We care about rigor, clarity, and impact. Disagreement is welcome when it’s thoughtful, the best idea wins regardless of who it comes from, and you’ll learn something new most weeks.

Your Impact

• Deploy and operate models in production: Run large language and multimodal models using inference engines like vLLM, SGLang, and TensorRT-LLM.

• Own the serving infrastructure: Build and maintain distributed inference systems with NVIDIA Dynamo, Kubernetes, Helm, and Ansible, including upgrades, monitoring, troubleshooting, and reliability.

• Optimize performance: Tune parallelism strategies (TP/EP/DP/PP), batching, KV caching, routing, and scheduling to meet latency, throughput, and SLA targets.

• Benchmark and measure: Build reproducible benchmarks for latency (TTFT, TPOT, E2E), throughput, concurrency scaling, and system efficiency.

• Support diverse workloads: Enable tool calling, agentic and coding workflows, RAG, and long-context applications.

• Bring new tech into production: Evaluate new models, frameworks, and optimizations, and partner with hardware, model, and systems engineers to ship improvements.

Qualifications

• BS, MS, or PhD in Computer Science, Computer Engineering, or a related field, or equivalent practical experience.

• 2+ years of engineering experience in LLM inference, ML systems, GPU systems, or related infrastructure.

• Hands-on experience deploying and optimizing at least one LLM inference engine (e.g., vLLM, SGLang, or TensorRT-LLM).

• Solid understanding of LLM inference fundamentals: prefill/decode, batching, KV caching, and latency/throughput tradeoffs.

• Strong Python and Linux skills, plus hands-on experience with Kubernetes and containerized deployments.

• Comfortable benchmarking and troubleshooting systems, and communicating clearly in a small engineering team.

Nice to Have

• Experience with Helm, Ansible, or similar deployment and configuration tools.

• Experience with distributed inference systems like NVIDIA Dynamo, including parallelism strategies (TP/EP/DP/PP), prefill/decode disaggregation, and KV-cache-aware routing.

• Experience with advanced inference optimizations such as prefix caching, speculative decoding, and quantization.

• Experience analyzing LLM architectures and tuning deployments across different hardware platforms to meet SLAs.

• Contributions to vLLM, SGLang, TensorRT-LLM, or related open-source projects.

• Experience owning production infrastructure, ideally at an early-stage startup.

What We Offer

• Competitive salary with performance-based bonus and early-stage equity grant

• 100% employer-paid Health, Dental, and Vision coverage for you and your dependents

• 401(k) match with immediate vesting, and access to financial advisors to help you reach your financial goals

• 100% employer-paid Life, Disability, and AD&D insurance, plus a fitness stipend and wellness & mental health perks

• Generous PTO: 20 vacation days, 15 company holidays (including 3 floating days of your choosing)

• Daily lunch stipend

• Enterprise-level Claude & ChatGPT access with a generous token budget

• Well-equipped, sunny offices in Santa Clara, CA & Cambridge, MA with on-site parking and EV charging; on-site fitness center in Santa Clara; gym discounts near our Cambridge office

• Relocation assistance available to work in Santa Clara, CA or Boston, MA

• Visa sponsorship and relocation assistance to one of our office hubs

• A collaborative, continuous-learning environment with smart, dedicated colleagues building the next generation of high-performance computing architecture