Senior Inference SW Engineer

Hewlett Packard Enterprise; Durham (NC); On-site; USD 140,000 - 180,000; Full time

Benefits offered by this job

Health And Wellbeing Benefits

Personal And Professional Development

Job summary

Hewlett Packard Enterprise is seeking an experienced software engineer to design, implement, and operate major components of the LLM serving runtime and its Kubernetes orchestration layer in Durham, NC. You will focus on improving latency, throughput, GPU utilization, and distributed execution while evaluating new inference technologies.

The role requires strong knowledge of inference engines, Kubernetes, Go and Python, and the ability to debug and profile C++/CUDA workloads, along with

Qualifications

• 8+ years software engineering experience
• 1–2+ years on LLM inference runtimes or production model serving
• Degree in Computer Science or related field

Responsibilities

• Design major components of LLM serving runtime and its Kubernetes orchestration layer
• Improve latency, throughput, GPU utilization, and distributed execution
• Evaluate emerging inference technologies and tools
• Resolve customer issues and provide effective support
• Contribute through code reviews, mentoring, and strong engineering practices

Skills

LLM Inference

Inference Runtime

vLLM

SGLang

TensorRT-LLM

Continuous Batching

KV Cache Management

Kubernetes

Go

Python

C++

CUDA

Tensor Parallelism

Pipeline Parallelism

NCCL

GPU Profiling

Education

Bachelor’s degree in Computer Science or related field

Job description

Design, implement, and operate major components of the LLM serving runtime and its Kubernetes orchestration layer, improving latency, throughput, GPU utilization, and distributed execution. Evaluate emerging inference technologies, resolve customer issues, and contribute through code reviews, mentoring, and strong engineering practices.

Requirements: Requires at least 8 years of software engineering experience, including 1–2 or more years working directly on LLM inference runtimes or production model serving; a degree in Computer Science or a related field is also specified. Candidates should have strong knowledge of inference engines and internals, Kubernetes, Go and Python, and be able to debug and profile C++/CUDA workloads.

Key Skills: LLM Inference, Inference Runtime Engineering, vLLM, SGLang, TensorRT-LLM, Continuous Batching, KV Cache Management, Kubernetes, Go, Python, C++, CUDA, Tensor Parallelism, Pipeline Parallelism, NCCL, GPU Profiling

Benefits: Health And Wellbeing Benefits, Personal And Professional Development