Senior LLM Inference Engineer - Hybrid, Edge-to-Cloud

Hewlett Packard Enterprise; Spring (TX); Hybrid; USD 144,000 - 315,000; Full time

Job summary

Hewlett Packard Enterprise's Private Cloud AI organization seeks a Senior Software Engineer to advance the model runtime of HPE AI Essentials, the inference platform enabling enterprises to run large language models on owned infrastructure, including air-gapped and sovereign environments.

The role focuses on engine integration, batching, KV cache management, and distributed execution, in collaboration with inference teams and the Kubernetes stack.

Qualifications

• Minimum of 8 years of experience in Software Engineering
• Experience working directly on LLM inference runtimes or production model serving
• Degree in Computer Science or related field

Responsibilities

• Design, implement, and own major components of the LLM serving deployment, including engine integration, continuous batching, KV cache management and reuse, and quantized execution
• Partner with inference engineering teams and contribute to improving time-to-first-token, inter-token latency, throughput per GPU, and P95/P99 tail latency
• Build and operate distributed execution capabilities, including disaggregated prefill/decode, tensor and pipeline parallelism, and KV cache offload across GPU memory, host memory, and RDMA-attached storage
• Evaluate emerging runtimes, quantization schemes, speculative decoding, and mixture-of-experts serving, and make well-supported recommendations on adoption
• Contribute to the orchestration layer supporting the runtime, including model admission, GPU scheduling and partitioning, cache-aware request routing, and autoscaling
• Triage and resolve customer issues end-to-end, identifying root causes and improving systems and processes to prevent recurrence
• Provide insightful code and design reviews, mentor team members, and lead by example on engineering practices within the team

Skills

LLM inference

Go

Python

C++/CUDA

Kubernetes

NCCL

Debugging/Profiling

Latency optimization

Education

Degree in Computer Science or related field

Tools

vLLM

SGLang

TensorRT-LLM

TGI

NVIDIA NIM

Job description

Hewlett Packard Enterprise's Private Cloud AI organization seeks a Senior Software Engineer to advance the model runtime of HPE AI Essentials, the inference platform enabling enterprises to run large language models on owned infrastructure, including air-gapped and sovereign environments.

The role focuses on engine integration, batching, KV cache management, and distributed execution, in collaboration with inference teams and the Kubernetes stack.