In-Person Senior ML Lead: LLM Evaluation & Agent Systems

TikTok USDS Joint Venture; San Jose (CA); On-site; USD 187,000 - 438,000; Full time

Job summary

TikTok USDS Joint Venture LLC seeks a Senior Machine Learning Engineer to lead architecture and execution of a next-generation AI Risk & Compliance Intelligence Platform and Intelligent Data Agent Ecosystem. You will bridge prompt engineering with native model-driven reasoning, tackling offline/online evaluation and synthetic data generation for small-sample scenarios.

You will drive ML patterns, design scalable data workflows, and mentor teams across Data, Analytics, and SRE.

Qualifications

• 5+ years of software/ML engineering experience with a strong track record of designing, building, and deploying ML/LLM systems in production.
• Deep hands-on expertise with LLM architectures, fine-tuning, RAG, prompt tuning, and agentic frameworks.
• Demonstrated experience building robust LLM evaluation frameworks (LLM-as-a-judge, automated benchmark pipelines) and tackling data scarcity/imbalance problems.
• Strong coding skills alongside production experience with scalable data processing pipelines.

Responsibilities

• Design, build, and scale robust LLM evaluation architecture, including automated offline datasets, metric suites, and benchmarking pipelines to measure prompt and model performance.
• Innovate on LLM-driven synthetic data generation methodologies to tackle data scarcity bottlenecks and systematically evaluate long-tail/edge-case risk scenarios.
• Advance beyond criteria-based prompt engineering by extracting deep risk patterns directly from complex case data to build native, ML-driven decision models and reasoning abstractions.
• Architect, evaluate, and continuously optimize modular agent skills and tools tailored for Data Science, Analytics, Data Engineering, and SRE workflows to maximize tool-use accuracy, execution efficiency, and task success rates.
• Implement advanced evaluation methodologies (such as LLM-as-a-judge and multi-agent benchmarking) across both risk intelligence and agentic ecosystems.
• Provide technical leadership by establishing core ML system design patterns, benchmarking standards, and engineering best practices, while translating complex operational requirements into scalable machine learning solutions across cross-functional partnerships.

Skills

LLM architectures

Prompt tuning

RAG

Data bootstrapping

Production ML

Python

Job description

TikTok USDS Joint Venture LLC seeks a Senior Machine Learning Engineer to lead architecture and execution of a next-generation AI Risk & Compliance Intelligence Platform and Intelligent Data Agent Ecosystem. You will bridge prompt engineering with native model-driven reasoning, tackling offline/online evaluation and synthetic data generation for small-sample scenarios.

You will drive ML patterns, design scalable data workflows, and mentor teams across Data, Analytics, and SRE.