Staff Data Scientist: Agent Improvement & Quality Lead

Yalo

United States

Remote

USD 180,000 - 260,000

Full time

9 days ago

A complete application in a minute

Tailored resume and cover letter, ready to send.

Get past ATS filters

Benefits offered by this job

Unlimited PTO
Remote work
Flexible hours
Startup environment
International teamwork
Career growth opportunities

Job summary

Yalo is seeking a Staff Data Scientist to elevate our AI agents. You will own evaluation systems, traces, and error analysis, translating production findings into improvements for agent skills, harness, and customers' flows.

You will collaborate with Product, Engineering, Delivery and Customer Success to prove what moved the metric. You will write production-grade Python and TypeScript, mentor teammates, and help shape quarterly planning for agent quality.

Qualifications

  • BS or MS in CS, Stats, Applied Math, OR Economics.
  • 8+ years delivering data science solutions with measurable business impact.
  • Hands-on evaluating LLM-based systems or agents (eval design, LLM-as-judge, rubric design).
  • Expert in Python and SQL; TypeScript is a plus.
  • Strong production engineering habits: version control, code review, testing, CI/CD.
  • Experience with GenAI systems in production: prompt and tool design, agent frameworks, and observability tooling.
  • Ability to turn messy trace data into clear findings for technical and non-technical audiences.
  • Strong product and business sense and customer-outcome focus.

Responsibilities

  • Own the evals strategy for agents: pre-release testing, production monitoring, and judges.
  • Run continuous error analysis on agent traces at scale; identify failure patterns and impact.
  • Close the loop: turn findings into fixes and new evals, ship and measure results.
  • Improve agent skills, harness and CLI using trace evidence; collaborate with owning teams.
  • Partner with delivery and customer teams to improve real customer flows.
  • Calibrate and validate LLM judges against human review and set evidence bar.
  • Write production-quality Python and TypeScript with tests and CI.
  • Mentor data scientists and engineers on measurement, experimentation and evaluation.
  • Shape quarterly and annual planning for agent quality.

Skills

Python
SQL
LLM evaluation
Prompt design
Data analysis
CI/CD
TypeScript
MLOps awareness

Education

BS or MS in CS/Statistics/Applied Math/Economics

Tools

LangGraph
LangSmith
Vertex
SageMaker
MLflow

Job description

Yalo is seeking a Staff Data Scientist to elevate our AI agents. You will own evaluation systems, traces, and error analysis, translating production findings into improvements for agent skills, harness, and customers' flows.

You will collaborate with Product, Engineering, Delivery and Customer Success to prove what moved the metric. You will write production-grade Python and TypeScript, mentor teammates, and help shape quarterly planning for agent quality.

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Staff Data Scientist: Agent Improvement & Evaluation
Staff Data Scientist: Agent Improvement & Evaluation

Yalo Inc. • United States

Remote
USD 180,000 - 240,000
Unlimited PTO
Competitive rewards
Remote work
+3
Staff Data Scientist, Agent Improvement Loop
Staff Data Scientist, Agent Improvement Loop

Yalo • United States

Remote
USD 180,000 - 260,000
Unlimited PTO
Remote work
Flexible hours
+3
Staff Data Scientist, Agent Improvement Loop
Staff Data Scientist, Agent Improvement Loop

Yalo Inc. • United States

Remote
USD 180,000 - 240,000
Unlimited PTO
Competitive rewards
Remote work
+3
Member of Technical Staff, Agents
Member of Technical Staff, Agents

Clera • Palo Alto (CA)

On-site
USD 200,000 - 300,000
Senior AI Engineer — Quality & Agent Systems (Hybrid)
Senior AI Engineer — Quality & Agent Systems (Hybrid)

Glean • Mountain View (CA)

Hybrid
USD 200,000 - 280,000
Home office stipend
Education stipend
Wellness stipend
+1
Member of Technical Staff, Agents
Member of Technical Staff, Agents

Clera • United States

On-site
USD 200,000 - 300,000
Senior Staff Engineer - Autonomous AI Agents (Production)
Senior Staff Engineer - Autonomous AI Agents (Production)

Clera • Palo Alto (CA)

On-site
USD 200,000 - 300,000
Member of Technical Staff, Agents
Member of Technical Staff, Agents

Royper Royco LLC • Palo Alto (CA)

On-site
USD 200,000 - 300,000
Senior AI Engineer: Agent Quality & Evaluation
Senior AI Engineer: Agent Quality & Evaluation

Bliro • New Munich (MN)

Hybrid
USD 101,000 - 146,000
Equity stake
30 days PTO
Hybrid work (Munich/remot e)
+4
Staff Data Scientist: AI System Quality & Experiments
Staff Data Scientist: AI System Quality & Experiments

Syllo • United States

Remote
USD 120,000 - 165,000