Staff Software Engineer — LLM Inference Platform Lead

Zenlayer, Inc

United States

Remote

USD 180,000 - 240,000

Full time

14 days+

A complete application in a minute

Tailored resume and cover letter, ready to send.

Get past ATS filters

Job summary

Zenlayer, Inc. is seeking a Technical Lead for inference performance to own end-to-end Token Performance. You will reduce cost per million tokens and increase per-GPU goodput while meeting SLOs across models from billions to trillions of parameters.

The role requires deep knowledge of Transformer inference, kernel-level optimization, and orchestration boundaries. You will drive disaggregation, explore MoE parallelism, and optimize CUDA kernels across GPUs.

Qualifications

  • Deep understanding of Transformer inference compute vs memory bottlenecks.
  • Experience identifying and fixing bottlenecks in scheduling, kernel selection, or KV management.
  • CUDA kernel tuning and memory optimization with hands-on experience.
  • Bachelor’s degree in CS/EE; 6+ years in systems/perf engineering.

Responsibilities

  • Establish performance metrics and load-testing for token performance and throughput.
  • Tune kernel-level inference engines and optimize scheduling, prefill, and KV paths.
  • Define boundaries between inference engine and orchestration layers (Dynamo/llm-d).
  • Lead KV cache transfer disaggregation and autoscaling strategies.
  • Experiment MoE parallelism and wide-EP design; quantify trade-offs.

Skills

Transformer inference knowledge
CUDA kernel tuning
Nsight profiling
Performance engineering
GPU performance work

Education

Bachelor’s degree in CS/EE or related field

Tools

Nsight
TensorRT-LLM
vLLM
SGLang
Dynamo llm-d

Job description

Zenlayer, Inc. is seeking a Technical Lead for inference performance to own end-to-end Token Performance. You will reduce cost per million tokens and increase per-GPU goodput while meeting SLOs across models from billions to trillions of parameters.

The role requires deep knowledge of Transformer inference, kernel-level optimization, and orchestration boundaries. You will drive disaggregation, explore MoE parallelism, and optimize CUDA kernels across GPUs.

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Staff Software Engineer, LLM Inference Platform
Staff Software Engineer, LLM Inference Platform

Zenlayer, Inc • United States

Remote
USD 180,000 - 240,000
Senior AI Inference Performance Engineer
Senior AI Inference Performance Engineer

NVIDIA Corporation • Santa Clara (CA)

On-site
USD 184,000 - 357,000
Senior LLM Inference: GPU Kernel Optimization
Senior LLM Inference: GPU Kernel Optimization

NVIDIA Corporation • Austin (TX)

On-site
USD 184,000 - 287,500
Equity
Benefits
Principal Software Engineer, Inference
Principal Software Engineer, Inference

Hewlett Packard Enterprise • Fort Collins (CO)

Hybrid
USD 180,000 - 250,000
Health and wellbeing benefits
Professional development programs
Flexible work arrangements
Tech Lead: LLM Inference & High-Performance Systems
Tech Lead: LLM Inference & High-Performance Systems

wekatest • United States

Remote
USD 180,000 - 240,000
Medical insurance
Dental insurance
Vision insurance
+5
AI Inference Kernel Optimization Engineer
AI Inference Kernel Optimization Engineer

Vidman AI • United States

Remote
USD 140,000 - 210,000
Principal LLM Training Performance Engineer | Equity
Principal LLM Training Performance Engineer | Equity

NVIDIA Corporation • Santa Clara (CA)

On-site
USD 272,000 - 431,000
Equity
Benefits
Senior LLM Training Performance Engineer
Senior LLM Training Performance Engineer

NVIDIA Corporation • California

On-site
USD 184,000 - 357,000
Equity
Benefits package
Inclusive culture
Principal Software Engineer, Inference
Principal Software Engineer, Inference

Hewlett Packard Enterprise • Durham (NC)

On-site
USD 180,000 - 240,000
Health & Wellbeing
Personal & Professional Development
Senior Inference Performance Engineer, Agent-Driven Optimization
Senior Inference Performance Engineer, Agent-Driven Optimization

2100 NVIDIA USA • Santa Clara (CA)

Hybrid
USD 124,000 - 242,000