Senior Software Engineer - LLM Inference Runtime (Hybrid)

Hewlett Packard Enterprise Company; Spring (TX); Hybrid; USD 137,000 - 315,000; Full time

Job summary

Hewlett Packard Enterprise seeks a Senior Software Engineer to build and evolve the model runtime for the AI Essentials inference platform used by enterprises to operate large language models on customer-owned hardware, including air-gapped environments. Emphasis is on sustained execution efficiency, low tail latency, and high GPU utilization.

The role involves engine integration, batching, KV cache management, and distributed execution, with collaboration across inference teams and a Kubernetes

Qualifications

• Experience with LLM inference runtimes or production model serving.
• Strong understanding of batching, KV cache reuse, quantization, and speculative decoding.
• Proficiency in Go and Python; ability to read/debug C++/CUDA.

Responsibilities

• Design, implement, and own major components of the LLM serving deployment, including engine integration, continuous batching, KV cache management and reuse, and quantized execution.
• Partner with inference engineering teams to improve time-to-first-token, inter-token latency, throughput per GPU, and tail latency (P95/P99).
• Build and operate distributed execution capabilities, including disaggregated prefill/decode, tensor and pipeline parallelism, and KV cache offload across GPU memory, host memory, and RDMA-attached storage.
• Evaluate emerging runtimes and serving strategies and make recommendations on adoption.
• Contribute to orchestration layer including model admission, GPU scheduling, cache-aware routing, and autoscaling.
• Triage and resolve customer issues end-to-end and provide code/design reviews, mentoring the team.

Skills

LLM inference engines

Kubernetes

Go

Python

Nsight

debugging/profiling

GPU memory hierarchy

inference internals

Education

Degree in Computer Science

Tools

Nsight

Job description

Hewlett Packard Enterprise seeks a Senior Software Engineer to build and evolve the model runtime for the AI Essentials inference platform used by enterprises to operate large language models on customer-owned hardware, including air-gapped environments. Emphasis is on sustained execution efficiency, low tail latency, and high GPU utilization.

The role involves engine integration, batching, KV cache management, and distributed execution, with collaboration across inference teams and a Kubernetes