LLM Evaluation & User Insight PM - E‑Commerce AI

TikTok USDS Joint Venture; San Jose (CA); On-site; USD 149,000 - 246,000; Full time

Job summary

TikTok USDS Joint Venture LLC seeks a PM to shape LLM evaluation for the E-commerce Recommendation team within TikTok Shop. You will design evaluation methodologies, define core metrics, and perform user research to understand buyer and creator needs in AI-powered discovery and recommendations.

You will lead annotation, preference modeling, and RLHF workflows, and align with Recommendation, Search, and AI Discovery teams to turn insights into actionable model improvements that boost ranking and

Qualifications

• Bachelor's degree or above with 1+ years of Product Management, Data Product Management, or AI Quality/Evaluation experience.
• Hands-on experience or deep understanding of LLM evaluation methodologies and prompt engineering.
• Strong background in quantitative and qualitative analysis, metric design, and translating complex user research/data insights into actionable technical specs.
• Ability to drive alignment with cross-functional stakeholders (Downstream Recommendation PMs, ML Engineers, Data Science, Ops) in a fast-paced environment.
• Preferred: 3+ years of PM experience focusing on LLM/GenAI evaluation, AI Quality, UXR, Data Annotation Strategy, or ML applications.
• Preferred: Prior experience in E-Commerce, Search & Recommendation, or Conversational AI ecosystems.
• Technical familiarity with LLM training/fine-tuning workflows, RLHF, and AI safety SOPs.

Responsibilities

• Design and own comprehensive evaluation methodologies (automated, model-based, and human evaluation) to assess LLM capabilities in e-commerce contexts
• Define core quality metrics and establish standardized benchmark datasets.
• Conduct deep qualitative and quantitative user research to uncover buyer/creator needs, mental models, and pain points when interacting with AI-driven discovery and recommendation experiences.
• Lead human annotation, preference modeling (e.g., RLHF data strategies, pairwise comparisons), and red-teaming initiatives to systematically capture qualitative feedback and transform it into actionable model improvement signals.
• Act as the core strategic bridge connecting LLM quality insights with downstream consumers—specifically Recommendation, Search, and AI Discovery PM/Engineering teams.
• Provide continuous model performance diagnosis, failure‑mode analysis, and strategic recommendations to downstream recommendation teams to help optimize their multi-objective ranking and context‑aware delivery logic.

Skills

LLM evaluation

Prompt engineering

Cross-functional alignment

User research

Metrics design

Education

Bachelor's degree or above

Job description

TikTok USDS Joint Venture LLC seeks a PM to shape LLM evaluation for the E-commerce Recommendation team within TikTok Shop. You will design evaluation methodologies, define core metrics, and perform user research to understand buyer and creator needs in AI-powered discovery and recommendations.

You will lead annotation, preference modeling, and RLHF workflows, and align with Recommendation, Search, and AI Discovery teams to turn insights into actionable model improvements that boost ranking and