Senior LLM Inference Runtime Engineer (Kubernetes & GPU)

Hewlett Packard Enterprise

Durham (NC)

On-site

USD 140,000 - 180,000

Full time

4 days ago

Be an early applicant

Application generator

A complete application in a minute — tailored resume and cover letter, ready to send.

Get past ATS filters

Benefits offered by this job

Health And Wellbeing Benefits

Personal And Professional Development

Job summary

Hewlett Packard Enterprise is seeking an experienced software engineer to design, implement, and operate major components of the LLM serving runtime and its Kubernetes orchestration layer in Durham, NC. You will focus on improving latency, throughput, GPU utilization, and distributed execution while evaluating new inference technologies.

The role requires strong knowledge of inference engines, Kubernetes, Go and Python, and the ability to debug and profile C++/CUDA workloads, along with

Qualifications

8+ years software engineering experience
1–2+ years on LLM inference runtimes or production model serving
Degree in Computer Science or related field

Responsibilities

Design major components of LLM serving runtime and its Kubernetes orchestration layer
Improve latency, throughput, GPU utilization, and distributed execution
Evaluate emerging inference technologies and tools
Resolve customer issues and provide effective support
Contribute through code reviews, mentoring, and strong engineering practices

Skills

LLM Inference

Inference Runtime

vLLM

SGLang

TensorRT-LLM

Continuous Batching

KV Cache Management

Kubernetes

Python

C++

CUDA

Tensor Parallelism

Pipeline Parallelism

NCCL

GPU Profiling

Education

Bachelor’s degree in Computer Science or related field

Job description

The role requires strong knowledge of inference engines, Kubernetes, Go and Python, and the ability to debug and profile C++/CUDA workloads, along with

Get your free, confidential resume review.

or drag and drop your file here.