Senior Inference SW Engineer

Hewlett Packard Enterprise Development LP; Spring (TX); On-site; USD 140,000 - 315,000; Full time

Job summary

Hewlett Packard Enterprise is seeking a Senior Software Engineer to build and evolve the model runtime within HPE AI Essentials, focusing on low tail latency and high GPU utilization on customer-owned hardware. The role is hybrid with 2 days per week in an HPE office and flexibility for other sites in the US.

You will design major LLM deployment components, collaborate with inference teams, and help scale distributed execution with quantization and caching strategies.

Qualifications

• Minimum of 8 years of software engineering experience.
• 1-2+ years on LLM inference runtimes or production model serving.
• Degree in Computer Science or related field.

Responsibilities

• Design and own major components of the LLM serving deployment.
• Improve time-to-first-token and tail latency across GPUs.
• Develop distributed execution capabilities across memory hierarchies.
• Evaluate runtimes and quantization schemes and recommend adoption.
• Support runtime orchestration, GPU scheduling, and autoscaling.
• Troubleshoot customer issues end-to-end and mentor teammates.

Skills

LLM inference

Inference internals

GPU memory hierarchy

NCCL operations

Go programming

Python programming

Code profiling

Multi-tier debugging

Problem solving

Education

Bachelor's degree in Computer Science or related field

Tools

Kubernetes

Nsight

C++/CUDA

Job description

Senior Inference SW Engineer

This role has been designed as 'Hybrid' with a requirement that you will work on average 2 days per week from an HPE office.

Who We Are:

Hewlett Packard Enterprise is the global edge-to-cloud company advancing the way people live and work. We help companies connect, protect, analyze, and act on their data and applications wherever they live, from edge to cloud, so they can turn insights into outcomes at the speed required to thrive in today’s complex world. Our culture thrives on finding new and better ways to accelerate what’s next. We know varied backgrounds are valued and succeed here. We have the flexibility to manage our work and personal needs. We make bold moves, together, and are a force for good. If you are looking to stretch and grow your career our culture will embrace you. Open up opportunities with HPE.

Job Description:

HPE's Private Cloud AI organization is seeking a Senior Software Engineer to build and evolve the model runtime within HPE AI Essentials, the inference platform used by enterprises to operate large language models on infrastructure they own, including air-gapped and sovereign environments. The core engineering challenge in this domain is not model deployment but sustained execution efficiency: achieving low tail latency and high GPU utilization on customer-owned hardware of varying generation and configuration.

The primary work location is as listed, but could be any other HPE site location in the US; however, remote work options will be considered.

Responsibilities

• Design, implement, and own major components of the LLM serving deployment, including engine integration, continuous batching, KV cache management and reuse, and quantized execution
• Partner with inference engineering teams and contribute to improving time-to-first-token, inter-token latency, throughput per GPU, and P95/P99 tail latency
• Build and operate distributed execution capabilities, including disaggregated prefill/decode, tensor and pipeline parallelism, and KV cache offload across GPU memory, host memory, and RDMA-attached storage
• Evaluate emerging runtimes, quantization schemes, speculative decoding, and mixture-of-experts serving, and make well-supported recommendations on adoption
• Contribute to the orchestration layer supporting the runtime, including model admission, GPU scheduling and partitioning, cache-aware request routing, and autoscaling
• Triage and resolve customer issues end-to-end, identifying root causes and improving systems and processes to prevent recurrence
• Provide insightful code and design reviews, mentor team members, and lead by example on engineering practices within the team

Knowledge and Skills Required

• Familiar with LLM inference engines such as vLLM, SGLang, TensorRT-LLM, TGI, or NVIDIA NIM, including modification of engine internals
• Strong understanding of inference internals, including continuous batching, paged attention, KV cache reuse and prefix caching, chunked prefill, quantization, and speculative decoding
• Working knowledge of tensor and pipeline parallelism, NCCL collective operations, and the GPU memory hierarchy and interconnect characteristics that govern them
• Advanced proficiency in Kubernetes platform architectures, including operators, custom resources, controllers, and scheduling
• Strong programming proficiency in Go and Python, with the ability to read, debug, and profile C++/CUDA using tools such as Nsight
• Familiar with debugging/profiling multi-tier application workloads such as RAG, Agents
• Excellent analytical, debugging, and problem-solving abilities

Preferred

• Upstream contribution to vLLM, SGLang, TensorRT-LLM, llm-d, LMCache, or KServe
• Disaggregated prefill/decode serving, or KV cache offload and reuse at scale
• RDMA, GPUDirect Storage, InfiniBand, or RoCE
• MIG, fractional GPU allocation, and multi-tenant GPU isolation
• On-premises, air-gapped, or regulated enterprise software delivery

Experience and Education

• Minimum of 8 years of experience in Software Engineering, including 1-2+ years working directly on LLM inference runtimes or production model serving
• Degree in Computer Science or related field

Accessibility

HPE is committed to creating an inclusive and accessible workplace and encourages applications from all qualified individuals, including those with disabilities. If you believe you require accommodation during any stage of the application or interview process, please submit your request by completing our secure form linked here.

Information about employee benefits offered in the US can be found at https://myhperewards.com/main/new-hire-enrollment.html

What We Can Offer You:

Health & Wellbeing We strive to provide our team members and their loved ones with a comprehensive suite of benefits that supports their physical, financial and emotional wellbeing.

Personal & Professional Development We also invest in your career because the better you are, the better we all are. We have specific programs catered to helping you reach any career goals you have — whether you want to become a knowledge expert in your field or apply your skills to another division.

Unconditional Inclusion We are unconditionally inclusive in the way we work and celebrate individual uniqueness. We know varied backgrounds are valued and succeed here. We have the flexibility to manage our work and personal needs. We make bold moves, together, and are a force for good.

Job:

Engineering Job Level: TCP_04

Salary

The expected salary/wage range for this position is provided below. Actual offer may vary from this range based upon geographic location, work experience, education/training, and/or skill level. – United States of America: Annual Salary USD 144,000 - 273,000 in Colorado // 137,000 - 315,000 in North Carolina & Texas The listed salary range reflects base salary. Variable incentives may also be offered.

Equal Employment Opportunity

HPE is an Equal Employment Opportunity/ Veterans/Disabled/LGBT employer. We do not discriminate on the basis of race, gender, or any other protected category, and all decisions we make are made on the basis of qualifications, merit, and business need. Our goal is to be one global team that is representative of our customers, in an inclusive environment where we can continue to innovate and grow together.

Equal Employment Opportunity (EEO)

Hewlett Packard Enterprise provides equal employment opportunity to any employee or applicant without regard to sex, gender, color, race, ethnicity, religion, creed, national origin, ancestry, citizenship, age, marital status, sexual orientation, gender identity and expression, physical or mental disability, medical condition, pregnancy, protected veteran status, uniformed service status, familial status, genetic information, political affiliation, or any other characteristic protected by federal, state, or local law.

Equal Employment Opportunity is the Law Equal Employment Opportunity is the Law - Supplement E-Verify (US & PR only)

HPE is an E-Verify employer. E-Verify is an Internet-based system that compares information from an employee's Form I-9, Employment Eligibility Verification, to data from U.S. Department of Homeland Security and Social Security Administration records to confirm the employment eligibility of all newly hired employees.

Disclosure of Sensitive Personal Data

Please ensure the resume you submit to us does not include any sensitive personal data. Sensitive personal data includes data revealing information about your racial or ethnic origin, political opinions, religious or philosophical beliefs, trade union membership, health, sex life or sexual orientation. To the extent the resume you submit does contain this type of personal data, you consent to the storing and processing of this data by HPE for the purpose of reviewing and managing your application.