Tech Lead Machine Learning Engineer - LLM Evaluation & Agent Systems

TikTok USDS JV; Seattle (WA); On-site; USD 198,000 - 416,000; Full time

Benefits offered by this job

Medical, dental, and vision insurance

401(k) with company match

Paid parental leave

Short/long-term disability coverage

Life insurance

Wellbeing benefits

10 paid holidays

10 paid sick days

Paid Personal Time

Job summary

TikTok USDS JV is seeking a Senior Machine Learning Engineer to lead the design and delivery of an AI Risk & Compliance Intelligence Platform and an Intelligent Data Agent Ecosystem. This onsite Seattle role combines architecture and hands-on execution across LLM evaluation, synthetic data generation, and modular agent systems for Data, Engineering, and SRE workflows.

You will work with the USDS team focusing on security, integrity, and compliance of TikTok's data in the United States,

Qualifications

• 5+ years of software or ML engineering experience
• Hands-on with LLM architectures, fine-tuning, RAG, prompt tuning
• Experience building robust LLM evaluation frameworks
• Strong coding skills for scalable data processing pipelines
• Prior domain experience in Risk & Compliance Intelligence
• Hands-on with synthetic data generation or bootstrapping models
• Experience building AI tools/agents for data personas
• Founding technical lead on high-impact initiatives

Responsibilities

• Design, build, and scale LLM evaluation architecture with datasets, metrics, and benchmarks.
• Develop LLM-driven synthetic data generation for risk scenarios.
• Create ML-driven decision models and reasoning abstractions from complex data.
• Architect and optimize modular agent skills and tools for Data Science, Analytics, Data Engineering, and SRE.
• Apply advanced evaluation methodologies like LLM-as-a-judge and multi-agent benchmarking.
• Provide technical leadership by defining ML system design patterns and best practices.

Skills

LLM architectures

Prompt tuning

LLM evaluation

Production ML

Python/ML pipelines

Tools

LLM frameworks

Job description

TikTok USDS JV is looking for a Senior Machine Learning Engineer technical leader to help build an AI Risk & Compliance Intelligence Platform and an Intelligent Data Agent Ecosystem. This onsite role in Seattle, WA combines architecture and execution across LLM evaluation, synthetic data generation, and modular agent systems that support Data, Engineering, and SRE workflows.

You will work within the TikTok U.S. Data Security (USDS) team, which focuses on the security, integrity, and compliance of TikTok's data in the United States. In parallel, collaboration with the Data Platform Team will support scalable, reliable, and cost-efficient infrastructure for self-service tools and scientific insights.

Responsibilities

• Design, build, and scale robust LLM evaluation architecture, including automated offline datasets, metric suites, and benchmarking pipelines to assess prompt and model performance.
• Develop LLM-driven synthetic data generation approaches to address data scarcity bottlenecks and enable systematic evaluation of long-tail and edge-case risk scenarios.
• Move beyond criteria-based prompt engineering by extracting risk patterns from complex case data to create native, ML-driven decision models and reasoning abstractions.
• Architect, evaluate, and continuously optimize modular agent skills and tools tailored for Data Science, Analytics, Data Engineering, and SRE workflows, improving tool-use accuracy, execution efficiency, and task success rates.
• Apply advanced evaluation methodologies (including LLM-as-a-judge and multi-agent benchmarking) across both risk intelligence and agentic ecosystems.
• Provide technical leadership by defining core ML system design patterns, benchmarking standards, and engineering best practices, and by translating complex operational requirements into scalable ML solutions with cross-functional partners.

Requirements

• 5+ years of software or ML engineering experience, including a strong track record designing, building, and deploying ML/LLM systems in production.
• Hands‑on expertise with LLM architectures, fine‑tuning, RAG, prompt tuning, and agentic frameworks.
• Experience building robust LLM evaluation frameworks (LLM-as-a-judge, automated benchmark pipelines) and working on data scarcity or imbalance problems.
• Strong coding skills with production experience supporting scalable data processing pipelines.
• Prior domain experience in Risk & Compliance Intelligence and Automated Decision Systems.
• Hands‑on experience with synthetic dataset generation or bootstrapping models for cold‑start scenarios.
• Experience building AI tools or agents for technical personas including Data Analysts, Data Scientists, Data Engineers, and Data SRE.
• Demonstrated ability to serve as a founding technical lead on ambiguous, high-impact initiative pods.

Location and Compensation

• Location: Seattle, WA (onsite)
• Salary: USD 198,360 - 416,100 per year

Benefits

• Day one access to medical, dental, and vision insurance
• 401(k) savings plan with company match
• Paid parental leave
• Short-term and long-term disability coverage
• Life insurance
• Wellbeing benefits, among others
• 10 paid holidays per year
• 10 paid sick days per year
• 17 days of Paid Personal Time (prorated upon hire with increasing accruals by tenure)

About USDS

TikTok USDS Joint Venture LLC is dedicated to the safety and security of millions of Americans who create, discover, and connect with what they love on the apps we operate. On-site presence across teams is intended to improve speed, alignment, and agility, particularly for real-time decision-making, team development, and integrated execution. The company is shifting from a hybrid work model to a fully in-person schedule up to 5 days a week.

USDS Reasonable Accommodation

USDS is committed to providing reasonable accommodations in its recruitment processes for candidates with disabilities, pregnancy, sincerely held religious beliefs, or other reasons protected by applicable laws.