Senior LLM Inference Engineer — Hybrid & Impactful

HITEC; Spring, Northern (TX, KY); Hybrid; USD 137,000 - 315,000; Full time

Job summary

Hewlett Packard Enterprise is seeking a Senior Software Engineer to advance the model runtime within HPE AI Essentials. You'll design and implement core components of the inference platform, including engine integration, batching, KV cache management, and distributed execution, with Kubernetes orchestration support.

The role emphasizes low tail latency and high GPU utilization on customer hardware across generations.

Qualifications

• Minimum of 8 years of experience in Software Engineering, including 1-2+ years working directly on LLM inference runtimes or production model serving.
• Degree in Computer Science or related field.
• Strong debugging and profiling skills across multi‑tier workloads.

Responsibilities

• Design, implement, and own major components of the LLM serving deployment, including engine integration, continuous batching, KV cache management and reuse, and quantized execution
• Partner with inference engineering teams to improve latency, throughput, and tail latency
• Build and operate distributed execution capabilities across GPU memory, host memory, and storage
• Evaluate emerging runtimes and quantization schemes and provide informed recommendations
• Contribute to orchestration layer, including GPU scheduling and autoscaling
• Triage customer issues end-to-end and mentor team members on engineering practices
• Provide code and design reviews and lead by example in best practices

Skills

LLM inference

Go

Python

Kubernetes

C++/CUDA

NCCL

Education

Degree in Computer Science or related field

Tools

Nsight

Docker

KServe

Job description

Hewlett Packard Enterprise is seeking a Senior Software Engineer to advance the model runtime within HPE AI Essentials. You'll design and implement core components of the inference platform, including engine integration, batching, KV cache management, and distributed execution, with Kubernetes orchestration support.

The role emphasizes low tail latency and high GPU utilization on customer hardware across generations.