Senior LLM Inference Engineer - Hybrid, Impact at Scale

Hewlett Packard Enterprise Company; Spring (TX); Hybrid; USD 137,000 - 315,000; Full time

Benefits offered by this job

Health & Wellbeing

Professional development

Inclusive culture

Job summary

Hewlett Packard Enterprise is seeking a Senior Inference Software Engineer to design and evolve the model runtime for enterprise AI platforms. The role focuses on low tail latency, high GPU utilization, and scalable distributed execution across heterogeneous hardware.

The position offers hybrid work with an average of two days per week in an HPE office in the United States, with remote options considered. Strong expertise in Go, Python, Kubernetes, and LLM inference is required.

Qualifications

• Minimum of 8 years of experience in Software Engineering
• 1-2+ years working directly on LLM inference runtimes or production model serving

Responsibilities

• Design, implement, and own major components of the LLM serving deployment.
• Partner with inference engineering teams to improve latency and throughput.
• Build and operate distributed execution capabilities across GPU memory and storage.
• Evaluate emerging runtimes and quantization schemes for adoption.
• Contribute to orchestration layer including GPU scheduling and autoscaling.
• Triage and resolve customer issues end-to-end and improve processes.
• Provide code/design reviews and mentor team members.

Skills

LLM inference

Go

Python

Kubernetes

NCCL

GPU memory hierarchy

Nsight

Profiling

Education

Bachelor's degree in Computer Science or related field

Tools

Nsight

KServe

TensorRT-LLM

RDMA

Job description

Hewlett Packard Enterprise is seeking a Senior Inference Software Engineer to design and evolve the model runtime for enterprise AI platforms. The role focuses on low tail latency, high GPU utilization, and scalable distributed execution across heterogeneous hardware.

The position offers hybrid work with an average of two days per week in an HPE office in the United States, with remote options considered. Strong expertise in Go, Python, Kubernetes, and LLM inference is required.