Senior ML Infra Engineer, Real-Time Recommendations & Search

TikTok USDS Joint Venture; Seattle (WA); On-site; USD 178,000 - 342,000; Full time

Benefits offered by this job

Restaurant

Job summary

TikTok USDS Joint Venture LLC is seeking an experienced ML engineer to own and advance real-time training and inference platforms for the TikTok recommendations and search engines. You’ll drive multi-node distributed systems, optimized serving, and secure data processing in collaboration with research and platform teams.

The role emphasizes scalable architecture, 3D parallelism, fault tolerance, and a strong focus on performance and reliability across massive GPU clusters.

Qualifications

• Bachelor's or Master's degree in Computer Science, Computer Engineering, or a related technical discipline.
• 4+ years of professional software engineering experience with Python and C++/Java.
• 2+ years building ML infrastructure at enterprise scale.
• Deep familiarity with PyTorch/TensorFlow and low-level GPU memory management, CUDA, and networking (InfiniBand/RoCE).
• Production-grade LLM training/inference tools and profiling to remove bottlenecks.
• Strong system troubleshooting and debugging skills.

Responsibilities

• Contribute to the technical roadmap and implement large-scale distributed real-time ML training and inference platforms.
• Design and scale multi-node distributed training with 3D parallelism (Data, Tensor, Pipeline).
• Build low-latency, high-throughput model serving infrastructure for massive traffic.
• Co-design next-gen generative recommendation systems with Applied ML Research teams.
• Develop automated fault-detection and checkpointing for multi-GPU clusters.
• Collaborate with data platform teams to enable high-throughput secure data processing.

Job description

TikTok USDS Joint Venture LLC is seeking an experienced ML engineer to own and advance real-time training and inference platforms for the TikTok recommendations and search engines. You’ll drive multi-node distributed systems, optimized serving, and secure data processing in collaboration with research and platform teams.

The role emphasizes scalable architecture, 3D parallelism, fault tolerance, and a strong focus on performance and reliability across massive GPU clusters.