Senior Data Scientist: AI Evaluation & LLM Judging

OpenTalent; Fremont (CA); On-site; USD 230,000 - 300,000; Full time

Job summary

Walmart’s Next Gen Commerce team is shaping the future of conversational shopping by building intelligent agents. A Distinguished Data Scientist for Quality & LLM Judging Systems will lead model development for evaluation methodologies, ensuring high-quality judgments for AI-powered conversations and tools.

You will design prompts, validate agreement with human judgment, and collaborate with product and platform teams to drive trustworthy AI and reliable metrics across the system.

Qualifications

• 7+ years of experience in data science or machine learning, preferably in evaluation, NLP, or conversational AI.
• Hands-on experience with large language models, including prompt engineering, response grading, and structured generation tasks.
• Familiarity with both human annotation workflows and automated evaluation strategies using LLMs.
• Deep understanding of metric design, evaluation reliability, and statistical validity.
• Strong software engineering fundamentals and ability to own end-to-end pipelines.

Responsibilities

• Design evaluation pipelines for conversational agents and their tool outputs using LLM-as-a-judge, human annotation, and hybrid methods.
• Develop high-quality prompts for structured evaluation tasks and iterate based on inter-rater reliability with human judges.
• Develop novel techniques to assess non-textual or subjective outputs—such as recommendations, summaries, and agent-driven actions—where standard metrics fall short.
• Guide the modeling team to distill or fine-tune smaller LLMs to act as scalable evaluation proxies.
• Work with engineering partners to integrate evaluation hooks into model training, validation, and production workflows.
• Conduct in-depth failure mode analysis and define actionable quality signals that inform model and production iteration.
• Uphold statistical rigor in metric design, validation, and experimental analysis to ensure reliable and interpretable results.
• Foster a culture of principled measurement and trustworthy AI throughout the organization.
• 7+ years of experience in data science or machine learning, preferably in evaluation, NLP, or conversational AI.
• Hands-on experience with large language models, including prompt engineering, response grading, and structured generation tasks.
• Familiarity with both human annotation workflows and automated evaluation strategies using LLMs.
• Deep understanding of metric design, evaluation reliability, and statistical validity.
• Strong software engineering fundamentals and ability to own end-to-end pipelines.

Job description

Walmart’s Next Gen Commerce team is shaping the future of conversational shopping by building intelligent agents. A Distinguished Data Scientist for Quality & LLM Judging Systems will lead model development for evaluation methodologies, ensuring high-quality judgments for AI-powered conversations and tools.

You will design prompts, validate agreement with human judgment, and collaborate with product and platform teams to drive trustworthy AI and reliable metrics across the system.