Product Manager, LLM Evaluation & UX Insights

TikTok USDS Joint Venture; Seattle (WA); On-site; USD 142,000 - 234,000; Full time

Job summary

TikTok USDS Joint Venture LLC is seeking a Product/AI Quality professional to design evaluation frameworks for LLM-powered e-commerce discovery and to align quality insights with Recommendation, Search, and AI Discovery teams. You will lead RMHF data strategies, drive metric definitions, and conduct user research to improve AI-driven experiences.

The role emphasizes collaboration with ML Engineers, Data Science, and Ops, with a focus on scalable evaluation methods and real-world impact on

Qualifications

• Bachelor's degree or higher with 1+ years in Product Management, Data Product Management, or AI Quality/Evaluation.
• Hands-on experience with LLM evaluation methodologies and prompt engineering.
• Strong background in quantitative and qualitative analysis, metric design, and translating insights into actionable specs.
• Ability to drive alignment with cross-functional stakeholders in a fast-paced environment.

Responsibilities

• Design and own evaluation methodologies (automated, model-based, and human) to assess LLM capabilities in e-commerce.
• Define core quality metrics and establish standardized benchmark datasets.
• Conduct qualitative and quantitative user research to uncover buyer/creator needs and pain points in AI-driven discovery.
• Lead human annotation and RLHF data strategies to translate feedback into model improvements.
• Bridge LLM quality insights with downstream Recommendation, Search, and AI Discovery teams.
• Provide model diagnostics and recommendations to optimize ranking and context-aware delivery.

Skills

LLM evaluation

Prompt engineering

Quantitative analysis

Qualitative analysis

Cross-functional alignment

Communication

Education

Bachelor's degree or above

Tools

LLM training/fine-tuning workflows

RLHF

Job description

TikTok USDS Joint Venture LLC is seeking a Product/AI Quality professional to design evaluation frameworks for LLM-powered e-commerce discovery and to align quality insights with Recommendation, Search, and AI Discovery teams. You will lead RMHF data strategies, drive metric definitions, and conduct user research to improve AI-driven experiences.

The role emphasizes collaboration with ML Engineers, Data Science, and Ops, with a focus on scalable evaluation methods and real-world impact on