Graduate Backend Inference Engine Engineer

ByteDance; San Jose (CA); On-site; USD 128,000 - 256,000; Full time

Benefits offered by this job

Medical insurance

Dental insurance

Vision insurance

401(k) with company match

Paid parental leave

Disability coverage

Life insurance

Wellbeing benefits

10 paid holidays per year

10 paid sick days per year

Paid Personal Time

Job summary

ByteDance in San Jose seeks a Senior GPU Inference Engineer to drive the iteration of large model inference engines and optimize GPU memory, latency, and throughput across multi-card systems.

You will design distributed parallel solutions (TP/PP/sequence/MoE) and push performance across vLLM, TensorRT-LLM, and related frameworks while collaborating with cross-functional teams to scale cutting-edge AI capabilities for internal products.

Qualifications

• Bachelor's/Master's degree in Software Development, CS, or related technical discipline.
• Solid foundation in computer low-level knowledge; proficient in C/C++ and Python.
• Proficient in CUDA programming; familiar with GPU memory models and scheduling.
• Experience with deep learning operators, graph optimization, and memory optimization.
• Experience using GPU performance tools like Nsight/Profiler and profiling for bottlenecks.
• Familiar with large model inference frameworks and multi-card parallelism concepts.

Responsibilities

• Iterate architecture of large model inference engine and optimize GPU latency and throughput.
• Adapt to GPU/NPU hardware architectures and ensure high performance across devices.
• Lead design and optimization of distributed parallel solutions (TP/PP/sequence/MoE).
• Stay updated on global large model inference tech and benchmark against frameworks like vLLM and TensorRT-LLM.

Skills

C/C++

Python

CUDA

GPU hardware

Performance analysis

Education

Bachelor's/Master's degree in CS/Engineering

Tools

Nsight

Profiler

TensorRT-LLM

Job description

ByteDance in San Jose seeks a Senior GPU Inference Engineer to drive the iteration of large model inference engines and optimize GPU memory, latency, and throughput across multi-card systems.

You will design distributed parallel solutions (TP/PP/sequence/MoE) and push performance across vLLM, TensorRT-LLM, and related frameworks while collaborating with cross-functional teams to scale cutting-edge AI capabilities for internal products.