Machine Learning Infrastructure Engineer, Recommendations and Search Seattle Regular R&D Job ID: A119310

TikTok Economic Impact

Seattle (WA)

On-site

USD 180,000 - 240,000

Full time

4 days ago
Be an early applicant

A complete application in a minute

Tailored resume and cover letter, ready to send.

Get past ATS filters

Job summary

TikTok USDS Joint Venture LLC is seeking a Machine Learning Infrastructure Engineer to advance the recommendations and search platforms. You will own and implement large-scale distributed training and real-time inference pipelines across billions of parameters, enabling rapid experimentation and production-grade deployment.

You will collaborate with ML researchers and data platform teams to optimize GPU clusters, ensure security/compliance, and maintain low-latency serving for live traffic

Qualifications

  • Bachelor's or Master’s degree in CS/CE or related field.
  • 2+ years software engineering with Python, C++, or Java.
  • 1+ year building ML infra at enterprise scale with ML frameworks (PyTorch/TF).

Responsibilities

  • Contribute to the technical roadmap for large-scale distributed real-time ML training and inference platforms.
  • Design and scale multi-node distributed training with 3D parallelism (Data/Tensor/Pipeline).
  • Build and optimize low-latency, high-throughput model serving infrastructure.

Skills

Python
C++/Java
PyTorch
TensorFlow
CUDA
Distributed systems
Inference serving

Education

Bachelor’s or Master’s in CS/CE

Job description

Machine Learning Infrastructure Engineer, Recommendations and Search

Seattle Regular R&D Job ID: A119310

Responsibilities

About the Team

We are a group of applied machine learning engineers that focus on TikTok recommendations and search engineers powering multiple product areas such as For-You-Page (FYP), Live Streaming, Global E-commerce, Local Services and more. We are developing innovative algorithms and techniques to improve user engagement and satisfaction, converting creative ideas into business-impacting solutions. We are interested in and excited about pushing the envelope of State-of-the-Art (SOTA) large scale machine learning to solve various real-world problems.

What You’ll Do

  • Technical Execution & System Ownership: Contribute to the technical roadmap and hands-on implementation of our large-scale (in billions parameters) distributed real-time ML training and inferencing platforms that power the TikTok recommendation and search engines, Short Form Video (SFV) ecosystem. Have a direct business impact on Live, Global E-Commerce, Local Services and many business domains.
  • Large-Scale Parallelism Architecture: Help design and scale multi-node distributed training systems, implementing advanced 3D parallelism strategies (Data, Tensor, Pipeline) to maximize compute efficiency and model scalability. Participate in the architecture, scale-testing, and maintenance of massive distributed computing foundations, GPU cluster configurations, and orchestration pipelines to achieve high hardware utilization and cluster efficiency under established SLI/SLO frameworks.
  • Production Inference & Serving: Build and scale low-latency, high-throughput model serving infrastructure, optimizing inference pipelines and leveraging low-level execution paths to handle massive live traffic under strict boundary isolation.
  • Algorithm-Infra Co-Design: Partner closely with Applied ML Research teams to co-design and pioneer next-generation Generative Recommendation systems, abstracting general-purpose components to support advanced generative paradigms in production.
  • Resiliency & Fault Tolerance: Build robust, automated fault-detection systems and asynchronous checkpointing mechanisms to gracefully handle hardware drops, silent data corruption (SDC), or network-switch failures in multi-thousand GPU clusters.
  • Cross-Functional Collaboration: Partner with Applied ML researchers and data platform teams to engineer high-throughput, secure multi-modal data processing and storage engines that prevent compliance friction.
  • Security & Compliance Hardening: Implement and enforce strict encryption-at-rest/in-transit controls, access control lists (ACLs), and secure tenant isolation protocols across the entire compute stack to meet compliance objectives.
Qualifications

Minimum Qualifications

  • Bachelor’s or Master’s degree in Computer Science, Computer Engineering, or a related technical discipline.
  • 2+ years of professional software engineering experience with deep expertise in Python, C++/Java.
  • 1+ years of direct experience building and maintaining machine learning infrastructure at enterprise scale.
  • Deep technical familiarity with the internals of core ML frameworks (PyTorch / TensorFlow) and a strong understanding of low-level GPU memory management, CUDA interactions, and networking topologies (InfiniBand/RoCE).
  • Solid understanding of production-grade LLM training and inference tools, with hands-on profiling skills to eliminate I/O, compute, or network bottlenecks.
  • Strong system-level troubleshooting and debugging skills, with experience profiling and eliminating I/O, compute, or network bottlenecks.

Preferred Qualifications

  • Experience optimizing high-performance training loops to maximize Model Flops Utilization (MFU) through advanced communication-computation overlap and zero-bubble pipeline scheduling across large-scale distributed clusters using industry-standard frameworks (e.g., Megatron, DeepSpeed).
  • Proven track record of scaling LLM training or inference workloads across hundreds of GPUs, with deep familiarity in advanced serving techniques such as KV Cache management, Prefill-Decoding (PD) separation, and model quantization.
  • Experience working in highly regulated industries, sovereign cloud environments, or dealing with federal data security compliance frameworks.
  • Strong flavor in low-level kernel development and graph compilation, with experience in CUDA, Triton, Cutlass, TensorRT, or Triton Inference Server being a huge plus.
  • Active background or interest in keeping up with the latest industry breakthroughs in MLOps, MoE (Mixture of Experts) routing infrastructure, and specialized hardware optimization.
  • Strong communication skills, with the ability to collaborate effectively across team boundaries, draft clear technical design docs, and mentor team members.
Job Information

About USDS

TikTok USDS Joint Venture LLC is dedicated to the safety and security of millions of Americans who create, discover, and connect with what they love on the apps we operate. The Joint Venture has been established in compliance with the Executive Order signed by President Trump on September 25, 2025. Our foundation is a comprehensive data privacy and cybersecurity program we operate under defined safeguards to protect national security and secure U.S. user data, apps and the algorithm. We safeguard the U.S. content ecosystem, holding decision-making authority for trust and safety policies and moderation. USDS Joint Venture helps ensure Americans can continue to express their creativity, discover new hobbies and interests, and build thriving communities and businesses on a global scale.

On-site presence across teams allows the company to operate with greater speed, alignment, and agility - especially in areas like real-time decision-making, team development, and integrated execution. As such, the company is shifting from a hybrid work model to a fully in-person schedule up to 5 days a week.

Why Join Us

Inspiring creativity is at the core of TikTok's mission. Our innovative product is built to help people authentically express themselves, discover and connect – and our global, diverse teams make that possible. Together, we create value for our communities, inspire creativity and bring joy - a mission we work towards every day.

We strive to do great things with great people. We lead with curiosity, humility, and a desire to make impact in a rapidly growing tech company. Every challenge is an opportunity to learn and innovate as one team. We're resilient and embrace challenges as they come. By constantly iterating and fostering an "Always Day 1" mindset, we achieve meaningful breakthroughs for ourselves, our company, and our users. When we create and grow together, the possibilities are limitless. Join us.

Diversity & Inclusion

TikTok is committed to creating an inclusive space where employees are valued for their skills, experiences, and unique perspectives. Our platform connects people from across the globe and so does our workplace. At TikTok, our mission is to inspire creativity and bring joy. To achieve that goal, we are committed to celebrating our diverse voices and to creating an environment that reflects the many communities we reach. We are passionate about this and hope you are too.

USDS Reasonable Accommodation

USDS is committed to providing reasonable accommodations in our recruitment processes for candidates with disabilities, pregnancy, sincerely held religious beliefs or other reasons protected by applicable laws. If you need assistance or a reasonable accommodation, please reach out to us at https://tinyurl.com/USDS-RA.

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Machine Learning Infrastructure Engineer, Recommendations and Search San Jose Regular R&D Job ID: A251835
Machine Learning Infrastructure Engineer, Recommendations and Search San Jose Regular R&D Job ID: A251835

TikTok Economic Impact • San Jose (CA)

On-site
USD 140,000 - 200,000
Machine Learning Engineer, Search - USDS Seattle Regular R&D Job ID: A152878
Machine Learning Engineer, Search - USDS Seattle Regular R&D Job ID: A152878

TikTok Economic Impact • Seattle (WA)

On-site
USD 140,000 - 210,000
Machine Learning Engineer, AI Search - USDS San Jose Regular R&D Job ID: A161616A
Machine Learning Engineer, AI Search - USDS San Jose Regular R&D Job ID: A161616A

TikTok Economic Impact • San Jose (CA)

On-site
USD 140,000 - 190,000
Machine Learning Engineer Graduate (Recommendations and Search) - 2027 Start
Machine Learning Engineer Graduate (Recommendations and Search) - 2027 Start

TikTok Inc. • Seattle (WA)

On-site
USD 100,000 - 178,000
Medical Insurance
401(k) Match
Paid Leave
Machine Learning Engineer Graduate (Recommendations and Search) - 2027 Start (PhD)
Machine Learning Engineer Graduate (Recommendations and Search) - 2027 Start (PhD)

TikTok Inc. • San Jose (CA)

On-site
USD 137,000 - 259,000
Medical, Dental, Vision insurance
401(k) with company match
Paid parental leave
+6
Software Engineer, AI Infrastructure - USDS Seattle Regular R&D Job ID: A221313
Software Engineer, AI Infrastructure - USDS Seattle Regular R&D Job ID: A221313

TikTok Economic Impact • Seattle (WA)

On-site
USD 120,000 - 180,000
Machine Learning Engineer Graduate (Recommendations and Search) - 2027 Start (PhD)
Machine Learning Engineer Graduate (Recommendations and Search) - 2027 Start (PhD)

TikTok Inc. • Seattle (WA)

On-site
USD 130,000 - 246,000
Health insurance
401(k) plan with company match
Paid parental leave
+6
Machine Learning Engineer Graduate (Tech and Product, USDS) - 2027 Start
Machine Learning Engineer Graduate (Tech and Product, USDS) - 2027 Start

TikTok Inc. • San Jose (CA)

On-site
USD 123,000 - 187,000
Health insurance
Dental insurance
Vision insurance
+7
Machine Learning Engineer Graduate (Recommendations and Search) - 2027 Start
Machine Learning Engineer Graduate (Recommendations and Search) - 2027 Start

TikTok Inc. • San Jose (CA)

On-site
USD 123,000 - 187,000
Medical, dental, vision insurance
401(k) with company match
Parental leave
+6
Machine Learning Engineer Graduate (Tech and Product, USDS) - 2027 Start
Machine Learning Engineer Graduate (Tech and Product, USDS) - 2027 Start

TikTok Inc. • Seattle (WA)

On-site
USD 100,000 - 178,000
Health insurance
401(k) match
Paid parental leave
+6