Senior Software Engineer, Inference

Hewlett Packard Enterprise; Durham (NC); On-site; USD 150,000 - 210,000; Full time

Benefits offered by this job

Health benefits

Professional development programs

Flexible work arrangements

Job summary

Hewlett Packard Enterprise is seeking an experienced software engineer to design and implement core components of the LLM runtime, focusing on engine integration, batching, and KV cache management. You will partner with distributed teams to optimize latency, throughput, and execution across deployments.

The ideal candidate has 8+ years in software engineering with 1-2+ years in LLM inference runtimes, and strong expertise in Go, Python, Kubernetes, and deep engine internals.

Qualifications

• Requires at least 8 years of software engineering experience with 1-2+ years in LLM inference runtimes or production model serving.
• Strong proficiency in Go, Python, Kubernetes, and deep knowledge of inference engine internals.
• Experience optimizing latency, throughput, and distributed execution for AI workloads.

Responsibilities

• Design and implement core components of the LLM runtime, including engine integration and batching.
• Collaborate with engineering teams to optimize inference performance and KV cache management.

Skills

LLM inference

Go

Python

Kubernetes

C++

CUDA

TensorRT-LLM

vLLM

Tensor parallelism

Pipeline parallelism

Performance profiling

Distributed execution

Tools

Kubernetes

Python

Go

C++

CUDA

TensorRT-LLM

vLLM

Distributed inference tooling

Job description

Design and implement key components of the LLM runtime, including engine integration, batching, and KV cache management. Partner with engineering teams to optimize inference performance, including latency, throughput, and distributed execution capabilities.

Requirements: Requires at least 8 years of software engineering experience with 1-2+ years specifically in LLM inference runtimes or production model serving. Candidates must possess strong proficiency in Go, Python, Kubernetes, and deep knowledge of inference engine internals.

Key Skills: LLM inference, Kubernetes, Go, Python, C++, CUDA, vLLM, TensorRT-LLM, Continuous batching, KV cache management, Distributed execution, GPU optimization, Tensor parallelism, Pipeline parallelism, Performance profiling

Benefits: Health and wellbeing benefits, Professional development programs, Flexible work arrangements