AI Infrastructure Engineer

Netpreme; Santa Clara, Boston (CA, MA); On-site; USD 150,000 - 210,000; Full time

Benefits offered by this job

Health, Dental, and Vision coverage

401(k) match

Life, Disability and AD&D insurance

PTO 20 days + 15 holidays

Lunch stipend

EV charging at offices

Visa sponsorship

Job summary

Netpreme is seeking an AI Infrastructure Engineer to build and operate the serving infrastructure on Kubernetes, focusing on vLLM and SGLang. You will work on a small, high-autonomy team to deploy and optimize large language models and design parallelism strategies across GPU systems.

You will build benchmarks, profile bottlenecks, and collaborate with hardware and model engineers to bring models into production while shaping the next generation of AI infrastructure.

Qualifications

• 2+ years in LLM inference, ML systems, GPU systems, or performance engineering.
• Hands-on deploying and tuning vLLM and/or SGLang.
• Strong understanding of prefill vs decode, batching, KV cache, latency/throughput.

Responsibilities

• Deploy and optimize large language and multimodal models using vLLM or other engines.
• Design parallelism strategies across GPU systems.
• Build benchmarks for TTFT, throughput, and memory utilization.
• Profile bottlenecks in GPU compute, memory, and serving runtime.
• Collaborate with hardware/systems team to translate performance needs into architecture.
• Work with model/system engineers to productionize new models.

Skills

Python

LLM inference

GPU systems

Performance engineering

vLLM

SGLang

Distributed systems

Communication

Batching

Education

BS, MS, or PhD in CS/CE or related field

Tools

Kubernetes

PyTorch Profiler

Nsight Systems

TensorRT-LLM

FlashInfer

Job description

About the Role

We're looking for an AI Infrastructure Engineer to build and operate the serving infrastructure You will work hands-on with vLLM and SGLang on Kubernetes. This is a foundational infrastructure role on a small, high-autonomy team.

Essential Duties & Responsibilities

• Deploy and optimize large language and multimodal models using vLLM and SGLang or other inference engines.
• Design and evaluate TP/EP/DP/PP and hybrid parallelism strategies across GPU systems.
• Build reproducible benchmarks to evaluate TTFT, TPOT, throughput, concurrency scaling, GPU utilization, and memory utilization.
• Analyze model architecture and its serving implications, including attention, KV cache, MoE, long context, and speculative decoding.
• Tune vLLM and SGLang configurations such as continuous batching, max batched tokens, chunked prefill, prefix caching, KV-cache precision/capacity, speculative decoding, CUDA Graphs, and P/D disaggregation.
• Profile and diagnose bottlenecks across GPU compute, memory, communication, scheduling, and serving runtime.
• Compare deployment configurations and identify production operating points balancing latency, throughput, capacity, and stability.
• Work with model/system engineers to bring newly released models into production efficiently.
• Collaborate closely with our hardware/systems team (direct access to CTO-level technical leadership on a small team) to translate performance requirements into backend architecture decisions.
• Contribute to defining next-generation benchmarks and service requirements as workloads evolve - multi-turn coding, agentic pipelines, RAG, and other long-context use cases.

Qualifications

• BS, MS, or PhD in Computer Science, Computer Engineering, or a related field, or equivalent experience.
• 2+ years of relevant experience in LLM inference, ML systems, GPU systems, or performance engineering.
• Must have: hands-on experience deploying and performance-tuning vLLM and/or SGLang.
• Strong understanding of LLM inference fundamentals, including prefill vs. decode, batching, KV cache, latency/throughput trade-offs, and distributed GPU execution.
• Strong Python engineering skills.
• Working knowledge of inference-serving concepts: continuous batching, KV cache handling, quantization, and serving SLAs.
• Clear written and verbal communication skills to work effectively with a small, fully distributed team.

Preferred Qualifications (optional)

• Contributions to vLLM, SGLang, FlashInfer, TensorRT-LLM, etc.
• Experience with MoE / long-context model deployment.
• Experience with speculative decoding, prefix caching, P/D disaggregation, attention/KV optimization.
• Experience with Nsight Systems / PyTorch Profiler.
• Familiarity with Kubernetes / production GPU serving.
• Previous startup experience.

Compensation & Benefits

• Competitive salary with performance-based bonus and early-stage equity grant
• 100% employer-paid Health, Dental, and Vision coverage for you and your dependents
• 401(k) match with immediate vesting, and access to financial advisors to help you reach your financial goals
• 100% employer-paid Life, Disability, and AD&D insurance, plus a fitness stipend and wellness & mental health perks
• Generous PTO: 20 vacation days, 15 company holidays (including 3 floating days of your choosing)
• Daily lunch stipend
• Enterprise-level Claude & ChatGPT access with a generous token budget
• Well-equipped, sunny offices in Santa Clara, CA & Cambridge, MA with on-site parking and EV charging; on-site fitness center in Santa Clara; gym discounts near our Cambridge office
• Visa sponsorship and relocation assistance to one of our office hubs
• A collaborative, continuous-learning environment with smart, dedicated colleagues building the next generation of high-performance computing architecture

The Opportunity

• Impact: Humanity stands at the dawn of a new industrial revolution driven by AI - one with the potential to redefine how we live on this planet. We are tackling a fundamental challenge at the infrastructure layer: unlocking greater AI capability while dramatically improving efficiency. The work we do here compounds across state-of-the-art AI models, systems, and real-world applications.
• Timing: Breakthrough technology matters most when it meets the right time. Joining now means real ownership of the company and meaningful influence over product direction and execution. In this early-stage environment, your ideas shape the trajectory of the technology - not just its implementation. You’ll work from first principles, move quickly from insight to execution, and see your contributions directly reflected in what we build.
• Culture: You’ll work alongside a group of people who care deeply about rigor, clarity, and impact. We value thoughtful disagreement, fast learning, and intellectual fearlessness. This is a place where strong ideas shine, curiosity is encouraged, and growth is a daily practice - not a future promise.