Senior Principal Software Engineer, LLM Inference & GPU Optimization

People In AI; United States; Remote; USD 230,000 - 280,000; Full time

Benefits offered by this job

Bonus

Equity

Remote work within United States

Job summary

People In AI is hiring a Senior Principal Software Engineer to optimize production LLM inference systems. You will work directly with inference runtimes, CUDA kernels, GPU memory architecture, and serving infrastructure for large-scale commercial applications.

This role focuses on making models run faster and more cost-efficient, going beyond deployment to optimize runtime internals, scheduling, and memory behavior. Remote work is supported for candidates in the United States.

Qualifications

• Significant software engineering experience with strong C++ fundamentals.
• Hands-on experience developing, modifying, or optimizing CUDA or similar GPU kernels.
• Experience with production LLM inference systems at meaningful scale.
• Deep understanding of GPU memory hierarchy, bandwidth, and bottlenecks.
• Experience profiling and improving latency, throughput, or GPU utilization.

Responsibilities

• Profile and optimize production LLM inference workloads for latency, throughput, and cost.
• Modify and improve inference runtimes such as vLLM and TensorRT LLM.
• Develop and optimize CUDA kernels and memory operations.
• Improve KV cache management, memory allocation, and inference performance under concurrency.
• Explore techniques like quantization and disaggregated inference; benchmark results in production.
• Collaborate with AI researchers and systems engineers across a global organization.

Skills

C++ programming

CUDA kernels

LLM inference systems

GPU memory architecture

Performance profiling

Tools

vLLM

TensorRT LLM

CUDA

Job description

Senior Principal Software Engineer, LLM Inference & GPU Optimization

Remote, United States

$230,000 to $280,000 base + bonus + potential equity

Work below the model API.

Most AI engineering roles focus on building applications around large language models. This one focuses on making the models themselves run faster, more efficiently, and at significantly lower cost.

We are partnering with an established global AI technology company to hire a Senior Principal Software Engineer to own and optimize production LLM inference systems.

You will work directly with inference runtimes, CUDA kernels, GPU memory architecture, and serving infrastructure supporting large-scale commercial applications.

This is not a traditional MLOps, model deployment, or LLM application engineering position. We need someone who understands what happens inside the inference engine and knows how to improve it.

What You'll Work On

• Profile and optimize production LLM inference workloads for latency, throughput, GPU utilization, and cost.
• Modify and improve inference runtimes such as vLLM, SGLang, and TensorRT LLM.
• Develop and optimize CUDA kernels, attention mechanisms, and GPU memory operations.
• Improve KV cache management, prefix caching, memory allocation, and inference performance under concurrent workloads.
• Optimize prefill and decode execution, including workloads with large input contexts and short generated responses.
• Explore techniques such as speculative decoding, continuous batching, quantization, and disaggregated inference.
• Identify performance bottlenecks, implement improvements, and benchmark measurable results in production.
• Collaborate with experienced AI researchers and systems engineers across a global engineering organization.

What We're Looking For

Your experience should go beyond deploying or configuring existing inference frameworks. You should have personally improved how production inference systems execute.

• Significant software engineering experience, with strong C++ programming fundamentals.
• Hands on experience developing, modifying, or optimizing CUDA or equivalent GPU kernels.
• Experience working with production LLM inference systems at meaningful scale.
• Strong understanding of GPU memory hierarchy, bandwidth, cache behavior, and compute bottlenecks.
• Experience profiling and improving latency, throughput, memory efficiency, or GPU utilization.
• Ability to navigate complex systems code, troubleshoot performance issues, and implement changes at the runtime level.
• Strong engineering judgment and the ability to explain technical decisions, tradeoffs, and measurable performance improvements.

Particularly Relevant Experience

• Deep involvement with vLLM, SGLang, TensorRT LLM, or comparable inference engines.
• FlashAttention or other attention kernel optimization.
• KV cache management, prefix reuse, and memory optimization.
• FP8, INT4, AWQ, GPTQ, or other quantization techniques.
• Prefill and decode disaggregation, speculative decoding, and inference scheduler optimization.
• NVIDIA H100 or H200 performance tuning.
• Rust development or distributed GPU inference systems.

Why Join?

Work on the problems that really move the needle. Your work will go beyond deploying models. You'll directly influence how efficiently they execute, improving the performance of commercially deployed AI systems.

Own meaningful technical challenges. Dive into GPU execution, memory behavior, inference scheduling, and runtime internals, with the scope to make substantial architectural and code-level improvements.

See measurable impact. Whether you're reducing inference latency, increasing tokens per second, or improving GPU efficiency, your engineering decisions will have tangible production outcomes.

Collaborate with exceptional technical talent. Join a globally distributed engineering team working across advanced AI research, high-performance software, and production infrastructure.

Enjoy flexibility and competitive rewards. Work remotely from anywhere in the United States, with comprehensive medical, dental, and vision coverage, retirement benefits, paid time off, bonus opportunities, and potential equity eligibility.

If you're the kind of engineer who wants to understand exactly why an LLM is slow—and then get into the code to make it faster, we'd love to hear from you.

About People In AI

People In AI is a specialist recruitment partner connecting exceptional AI, machine learning, and software engineering talent with companies building impactful AI technology.

We work closely with engineering leaders to identify opportunities where technical expertise can make a genuine difference.