ML Infra Engineer: Scalable Training & Real-Time Inference

TikTok USDS Joint Venture LLC; San Jose (CA); On-site; USD 140,000 - 200,000; Full time

Job summary

TikTok USDS Joint Venture LLC is hiring a Machine Learning Infrastructure Engineer for the Recommendations and Search domain in San Jose. You will own and implement scalable ML training and inference platforms powering TikTok’s recommendation and SFV ecosystems.

The role requires hands-on work with distributed GPU clusters, 3D parallelism, and low-latency serving, partnering with Applied ML and data platform teams to ensure secure, high-performance systems.

Qualifications

• 2+ years of professional software engineering experience with deep expertise in Python, C++/Java.
• 1+ years of direct experience building and maintaining ML infrastructure at enterprise scale.
• Deep technical familiarity with core ML frameworks (PyTorch/TensorFlow) and low-level GPU memory management, CUDA and networking topologies (InfiniBand/RoCE).
• Strong production-grade LLM training and inference tooling knowledge and profiling skills to mitigate I/O, compute, or network bottlenecks.

Responsibilities

• Contribute to the technical roadmap and hands-on implementation of large-scale distributed real-time ML training and inference platforms.
• Design and scale multi-node distributed training systems with 3D parallelism (Data, Tensor, Pipeline).
• Build and scale low-latency, high-throughput model serving infrastructure for massive live traffic.

Skills

Python

C++/Java

PyTorch/TensorFlow

Education

Bachelor's or Master's in CS/CE or related field

Tools

Megatron

DeepSpeed

CUDA

TensorRT

Triton Inference Server

Job description

TikTok USDS Joint Venture LLC is hiring a Machine Learning Infrastructure Engineer for the Recommendations and Search domain in San Jose. You will own and implement scalable ML training and inference platforms powering TikTok’s recommendation and SFV ecosystems.

The role requires hands-on work with distributed GPU clusters, 3D parallelism, and low-latency serving, partnering with Applied ML and data platform teams to ensure secure, high-performance systems.