Inference Systems Engineer - Large-Scale GPU & LLM

ByteDance; Singapore; On-site; SGD 180,000 - 240,000; Full time

Job summary

ByteDance Singapore is hiring to advance Volcano Engine's large-model training and inference systems, optimizing performance across thousands of GPUs and managing ultra-large-scale inference traffic. The role focuses on C/C++ and Python on Linux, with experience in distributed ML pipelines, and collaboration with algorithm teams to improve both algorithms and systems.

Expected to explore architectures, hardware integration (GPUs/TPUs), and perform heavy profiling and optimization in a

Qualifications

• Proficiency in C/C++ and Python on Linux for large-scale ML systems.
• Experience with at least one major ML framework (TF, PyTorch, MXNet).
• Familiar with large-model training/inference frameworks (vLLM, TensorRT-LLM, Megatron-LM).
• Strong problem-solving, independence, teamwork and communication.

Responsibilities

• Develop and optimize Volcano Engine large-model training and inference systems, scaling across thousands of GPUs and scheduling large-scale inference traffic.
• Solve challenges related to high concurrency, reliability and scalability for hundreds of billions of tokens.
• Research architectures for training/inference, including subgraph matching, compiler optimization, and model quantization.
• Integrate heterogeneous hardware (GPUs/NPUs/TPUs) with training and inference frameworks.
• Improve compute utilization across globally distributed GPU clusters through elastic scheduling and orchestration.
• Collaborate with algorithm teams to jointly optimize algorithms and systems.

Skills

C/C++

Python

Linux

distributed ML systems

TensorFlow

PyTorch

MXNet

Megatron-LM

vLLM

TensorRT-LLM

Education

Master's degree or PhD in computer science or related field

Job description

Location:

Team:

Employment Type:

Regular

Job Code:

A198864

Share this listing:

Responsibilities

About the TeamThe Applied Machine Learning (AML) - Enterprise team provides machine learning platform products on VolcanoEngine with cloud native resource scheduling system which intelligently orchestrates different tasks and jobs with minimised costs of every experiment and maximised resource utilisation, rich modelling tools including customised machine learning tasks and web IDE, and multi-framework high performance model inference services.In 2021, through VolcanoEngine, we released this machine learning infrastructure to the public, to provide more enterprises with reduced costs of computation power, lower barriers to machine learning engineering and deeper developments in AI capabilities.Responsibilities-Responsible for the development and performance optimization of the Volcano Engine large model training and inference systems, including but not limited to model computation optimization, tuning thousand-GPU training clusters, distributed LLM inference systems, and large-scale inference traffic scheduling.-Responsible for solving technical challenges related to high concurrency, high reliability, and high scalability, supporting Volcano Engine’s daily training and inference traffic at the scale of hundreds of billions of tokens.-Responsible for researching and introducing forward-looking technical architectures for large model training and inference, including but not limited to subgraph matching, compiler optimization, and model quantization.-Responsible for integrating heterogeneous hardware with training and inference frameworks, including but not limited to GPUs, NPUs, and TPUs.-Focused on improving compute utilization across globally distributed ultra-large-scale GPU clusters through elastic scheduling, GPU oversubscription, task orchestration, and related techniques.-Collaborate closely with algorithm teams to jointly optimize algorithms and systems.

Qualifications

Minimum Qualifications-Proficient in C/C++ and Python development under Linux environments, with experience in large-scale machine learning systems or search, advertising, and recommendation systems.-Familiar with at least one machine learning framework, such as TensorFlow, PyTorch, MXNet, or other in-house frameworks.-Familiar with at least one large model training or inference framework, including but not limited to vLLM, TensorRT-LLM, SGLang, and Megatron-LM.-Strong problem-solving skills with the ability to work independently, excellent teamwork spirit, and outstanding capability in breaking down complex problems.-Strong sense of responsibility, with good learning ability, communication skills, and self-motivation.Preferred Qualifications-Experience in designing architectures for large-scale distributed systems.-Strong understanding of GPU hardware architecture and the GPU software stack (such as CUDA and cuDNN), with experience in GPU performance analysis and optimization.-Research background at the Master’s or PhD level in computer systems–related fields, including distributed systems, parallel computing, programming languages and compilers, networking, or storage systems.

Job Information

About Us

Founded in 2012, ByteDance's mission is to inspire creativity and enrich life. With a suite of more than a dozen products, including TikTok, Lemon8, CapCut and Pico as well as platforms specific to the China market, including Toutiao, Douyin, and Xigua, ByteDance has made it easier and more fun for people to connect with, consume, and create content.

Why Join ByteDance

Inspiring creativity is at the core of ByteDance's mission. Our innovative products are built to help people authentically express themselves, discover and connect – and our global, diverse teams make that possible. Together, we create value for our communities, inspire creativity and enrich life - a mission we work towards every day.

As ByteDancers, we strive to do great things with great people. We lead with curiosity, humility, and a desire to make impact in a rapidly growing tech company. By constantly iterating and fostering an "Always Day 1" mindset, we achieve meaningful breakthroughs for ourselves, our Company, and our users. When we create and grow together, the possibilities are limitless. Join us.

Diversity & Inclusion

ByteDance is committed to creating an inclusive space where employees are valued for their skills, experiences, and unique perspectives. Our platform connects people from across the globe and so does our workplace. At ByteDance, our mission is to inspire creativity and enrich life. To achieve that goal, we are committed to celebrating our diverse voices and to creating an environment that reflects the many communities we reach. We are passionate about this and hope you are too.