Staff Data Scientist: Agent Improvement & Evaluation

Yalo Inc.

United States

Remote

USD 180,000 - 240,000

Full time

9 days ago

A complete application in a minute

Tailored resume and cover letter, ready to send.

Get past ATS filters

Benefits offered by this job

Unlimited PTO
Competitive rewards
Remote work
Flexible schedule
Startup environment
International teamwork

Job summary

Yalo is a fast-growing platform bringing conversational commerce to diverse markets. As a Staff Data Scientist, you shape evaluation strategies, monitor agent performance, and drive improvements across Python/TypeScript tooling and production traces.

You will partner with Product, Engineering, Delivery and Customer Success to translate findings into measurable impact. You will mentor teammates, help define quarterly plans for agent quality, and lead initiatives to improve the reliability of our

Qualifications

  • BS or MS in a technical field (CS/Stats/Applied Math/OR/Economics).
  • 8+ years delivering data science solutions with measurable business impact.
  • Hands-on experience evaluating LLM-based systems or agents.
  • Expert in Python and SQL. Comfortable in TypeScript.
  • Strong production engineering habits and CI/CD.

Responsibilities

  • Own the evals strategy for our agents: pre-release testing and production monitoring.
  • Run continuous error analysis on agent traces at scale and rank improvements.
  • Close the loop: turn findings into fixes, ship new evals, measure results.
  • Improve agent skills, harness, and CLI using trace evidence.
  • Partner with delivery and customer teams to improve real customer flows.
  • Calibrate and validate LLM judges against human review.
  • Write production-quality Python and TypeScript with tests and CI.
  • Mentor data scientists and engineers on measurement and evaluation.

Skills

Python
SQL
TypeScript
LLM evals
Cross-functional collab

Education

BS or MS in CS/Stats/Applied Math/OR/Economics

Tools

LangGraph
LangSmith

Job description

Yalo is a fast-growing platform bringing conversational commerce to diverse markets. As a Staff Data Scientist, you shape evaluation strategies, monitor agent performance, and drive improvements across Python/TypeScript tooling and production traces.

You will partner with Product, Engineering, Delivery and Customer Success to translate findings into measurable impact. You will mentor teammates, help define quarterly plans for agent quality, and lead initiatives to improve the reliability of our

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Staff Data Scientist: Agent Improvement & Quality Lead
Staff Data Scientist: Agent Improvement & Quality Lead

Yalo • United States

Remote
USD 180,000 - 260,000
Unlimited PTO
Remote work
Flexible hours
+3
Staff Data Scientist, Agent Improvement Loop
Staff Data Scientist, Agent Improvement Loop

Yalo • United States

Remote
USD 180,000 - 260,000
Unlimited PTO
Remote work
Flexible hours
+3
Staff Data Scientist, Agent Improvement Loop
Staff Data Scientist, Agent Improvement Loop

Yalo Inc. • United States

Remote
USD 180,000 - 240,000
Unlimited PTO
Competitive rewards
Remote work
+3
Member of Technical Staff, Agents
Member of Technical Staff, Agents

Clera • United States

On-site
USD 200,000 - 300,000
Member of Technical Staff, Agents
Member of Technical Staff, Agents

Clera • Palo Alto (CA)

On-site
USD 200,000 - 300,000
Member of Technical Staff, Agents
Member of Technical Staff, Agents

Royper Royco LLC • Palo Alto (CA)

On-site
USD 200,000 - 300,000
AI Engineer — Build Production Autonomous Agents
AI Engineer — Build Production Autonomous Agents

Yolk • Illinois

Hybrid
USD 120,000 - 150,000
Equity
Mentorship
AI Agent Developer (Latam)
AI Agent Developer (Latam)

Knowmad Mood • United States

Remote
USD 78,000 - 100,000
Staff Data Scientist, AI Agents Platform (Onsite)
Staff Data Scientist, AI Agents Platform (Onsite)

Walmart Global Tech • Bentonville (AR)

On-site
USD 140,000 - 180,000
401(k) match
Stock purchase plan
Paid maternity and parental leave
+2
Principal Data Scientist
Principal Data Scientist

Yoodli AI Roleplays • Seattle (WA)

Hybrid
USD 180,000 - 320,000
Competitive compensation
Meaningful equity
Hybrid work environment
+1