Large-Model Inference Runtime Engineer

Bytedance; San Jose (CA); On-site; USD 128,000 - 256,000; Full time

Job summary

ByteDance is seeking a highly skilled software engineer to advance its data AML mid-platform, focusing on large model inference and GPU optimization. The role emphasizes architecture iteration, end-to-end performance tuning, and cross-team collaboration across the Technology group in a fast-paced environment.

You will work on cutting-edge inference systems powering Douyin, Jinri Toutiao, and Xigua Video, with opportunities for growth and impact.

Qualifications

• Bachelor's or Master's degree in Software Development, Computer Science, Computer Engineering, or related technical discipline.
• Solid foundation in low-level computer knowledge; proficient in C/C++ and Python; CUDA programming; familiar with GPU hardware and memory models.
• Proficient in developing and optimizing deep learning operators; independent in operator reconstruction, memory access optimization, vectorization, and precision alignment.

Responsibilities

• Iterate the architecture of the large model inference engine and optimize GPU performance (fusion, compilation, memory, scheduling).
• Adapt to GPU/NPU hardware architectures; refine universality of the inference engine and hardware adaptability.
• Design, develop, and optimize distributed parallel solutions for large model inference (tensor/pipeline/sequence/MoE parallelism).
• Follow cutting-edge large model inference tech, benchmark against frameworks like vLLM and TensorRT-LLM; iterate to improve performance and cost.

Skills

C/C++

Python

CUDA programming

GPU architecture

Deep learning operators

GPU memory models

Nsight/Profiler

Cross-team collaboration

Performance optimization

Education

Bachelor's/Master's in CS/Engineering

Tools

Nsight

Profiler

Job description

ByteDance is seeking a highly skilled software engineer to advance its data AML mid-platform, focusing on large model inference and GPU optimization. The role emphasizes architecture iteration, end-to-end performance tuning, and cross-team collaboration across the Technology group in a fast-paced environment.

You will work on cutting-edge inference systems powering Douyin, Jinri Toutiao, and Xigua Video, with opportunities for growth and impact.