Staff Data Scientist, Agent Improvement Loop

Yalo

United States

Remote

USD 180,000 - 260,000

Full time

9 days ago

A complete application in a minute

Tailored resume and cover letter, ready to send.

Get past ATS filters

Benefits offered by this job

Unlimited PTO
Remote work
Flexible hours
Startup environment
International teamwork
Career growth opportunities

Job summary

Yalo is seeking a Staff Data Scientist to elevate our AI agents. You will own evaluation systems, traces, and error analysis, translating production findings into improvements for agent skills, harness, and customers' flows.

You will collaborate with Product, Engineering, Delivery and Customer Success to prove what moved the metric. You will write production-grade Python and TypeScript, mentor teammates, and help shape quarterly planning for agent quality.

Qualifications

  • BS or MS in CS, Stats, Applied Math, OR Economics.
  • 8+ years delivering data science solutions with measurable business impact.
  • Hands-on evaluating LLM-based systems or agents (eval design, LLM-as-judge, rubric design).
  • Expert in Python and SQL; TypeScript is a plus.
  • Strong production engineering habits: version control, code review, testing, CI/CD.
  • Experience with GenAI systems in production: prompt and tool design, agent frameworks, and observability tooling.
  • Ability to turn messy trace data into clear findings for technical and non-technical audiences.
  • Strong product and business sense and customer-outcome focus.

Responsibilities

  • Own the evals strategy for agents: pre-release testing, production monitoring, and judges.
  • Run continuous error analysis on agent traces at scale; identify failure patterns and impact.
  • Close the loop: turn findings into fixes and new evals, ship and measure results.
  • Improve agent skills, harness and CLI using trace evidence; collaborate with owning teams.
  • Partner with delivery and customer teams to improve real customer flows.
  • Calibrate and validate LLM judges against human review and set evidence bar.
  • Write production-quality Python and TypeScript with tests and CI.
  • Mentor data scientists and engineers on measurement, experimentation and evaluation.
  • Shape quarterly and annual planning for agent quality.

Skills

Python
SQL
LLM evaluation
Prompt design
Data analysis
CI/CD
TypeScript
MLOps awareness

Education

BS or MS in CS/Statistics/Applied Math/Economics

Tools

LangGraph
LangSmith
Vertex
SageMaker
MLflow

Job description

Yalo

Hi! This is Yalo! We are on a mission to bring conversational commerce to the world... Remember how it used to be to interact with businesses that knew and understood you, that could recommend exactly what you needed, and that with a simple message could get you what you wanted??? Yep... neither do we. That is why at Yalo we are marrying the scale of digital commerce with the personalization and simplicity of conversations to help companies delight their users. We know that traditional SAAS companies focus on first-world problems... we don't! Having started in Latin America, our roots are in Emerging Markets and therefore we care about bringing amazing experiences to a population that traditionally has been underserved, such as the small shop owner in Brazil who is ordering online for the first time. If you're looking for a place to make things happen, learn fast, and impact emerging markets in a way that hasn't been done before, look no further. Come Join us in our mission of improving billions of lives through the power of conversational commerce!

Your mission

As a Staff Data Scientist, you make our AI agents measurably better, and you prove it. You own how Yalo evaluates agents: evals, trace and error analysis, and the improvement loop that turns what we find in production into fixes to our agent skills, harness, CLI and customers' conversational flows. You work with Product, Engineering, Delivery and Customer Success. Your job is to show what moved the metric, not to guess.

What are the responsibilities for this role?
  • Own the evals strategy for our agents: pre-release behavioral testing, production monitoring, and the judges that score conversations.
  • Run continuous error analysis on agent traces at scale. Find failure patterns, attribute them to skills, prompts or releases, and rank improvement opportunities by impact.
  • Close the loop. Turn findings into fixes and new evals, ship them, and measure the result with sound before-and-after and causal methods.
  • Improve our agent skills, harness and CLI using trace evidence. Work through the owning teams' repos and review process.
  • Partner with delivery and customer teams to take real customer flows from underperforming to reliably healthy, and make the process repeatable for others.
  • Calibrate and validate LLM judges against human review. Measure their stability, report their limits, and set the bar for what counts as evidence at Yalo.
  • Write production-quality Python and TypeScript for evals and analysis tooling, with tests and CI, so it can live in a shared codebase.
  • Mentor data scientists and engineers on measurement, experimentation and evaluation practice.
  • Help shape quarterly and annual planning for agent quality.
Job Requirements (Must Have)
  • BS or MS in a technical field (Computer Science, Statistics, Applied Math, Operations Research, Economics, etc.).
  • 8+ years delivering data science solutions with measurable business impact in cross-functional teams.
  • Hands-on experience evaluating LLM-based systems or agents: eval design, LLM-as-judge, rubric design, and judge calibration.
  • Expert in Python and SQL. Comfortable in TypeScript, or ready to be.
  • Strong production engineering habits: version control, code review, testing, CI/CD, and shipping tools others use.
  • Experience with GenAI systems in production: prompt and tool design, agent frameworks (e.g., LangGraph), and LLM observability tooling (e.g., LangSmith).
  • Skilled at turning messy trace and conversation data into clear findings for technical and non-technical audiences.
  • Strong product and business sense. You frame quality work in terms of customer outcomes.
Nice To Have
  • Experience with conversational AI, customer-facing agents or tool-using agents.
  • NLP and clustering for failure-pattern discovery.
  • Causal inference and experimentation skills: A/B tests, difference-in-differences, pre/post analysis, and the judgment to say what the data can't show.
  • Exposure to MLOps or LLMOps platforms (Vertex, SageMaker, MLflow, or similar) and big data tools.
  • Familiarity with AI safety and responsible AI practices.
  • Comfort in an ambiguous, fast-changing environment.
  • Experience mentoring other data scientists.
What do we offer?
  • Unlimited PTO policy
  • Competitive rewards on the market range
  • Remote working is available (-+3 hours CT)
  • Flexible time (driven by results)
  • Start-up environment
  • International teamwork
  • You and nothing else limit your career here

We care, We keep it simple, We make it happen, We strive for excellence.

At Yalo, we are dedicated to creating a workplace that embodies our core values: caring, initiative, excellence, and simplicity. We believe in the power of diversity and inclusivity, where everyone's unique perspectives, experiences, and talents contribute to our collective success. As we embrace and respect our differences, we strive to create something extraordinary for the benefit of all.

We are proud to be an Equal Opportunity Employer, providing equal opportunities to individuals regardless of race, color, religion, national or ethnic origin, gender, sexual orientation, gender identity or expression, age, disability, protected veteran status, or any other legally protected characteristic. Our commitment to fairness and equality is a fundamental pillar of our company.

At Yalo, we uphold a culture of excellence. We constantly challenge ourselves to go above and beyond, delivering remarkable results and driving innovation. We encourage each team member to take initiative and make things happen, empowering them to bring their best ideas forward and contribute to our shared goals.

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Staff Data Scientist, Agent Improvement Loop
Staff Data Scientist, Agent Improvement Loop

Yalo Inc. • United States

Remote
USD 180,000 - 240,000
Unlimited PTO
Competitive rewards
Remote work
+3
Sales Engineer
Sales Engineer

Yalo Inc. • United States

Remote
USD 110,000 - 160,000
Unlimited PTO
Competitive rewards
Worklife integration
+2
Staff Data Scientist: Agent Improvement & Evaluation
Staff Data Scientist: Agent Improvement & Evaluation

Yalo Inc. • United States

Remote
USD 180,000 - 240,000
Unlimited PTO
Competitive rewards
Remote work
+3
Senior Data Analyst
Senior Data Analyst

Rightsline • United States

On-site
USD 120,000 - 160,000
Staff Data Scientist: Agent Improvement & Quality Lead
Staff Data Scientist: Agent Improvement & Quality Lead

Yalo • United States

Remote
USD 180,000 - 260,000
Unlimited PTO
Remote work
Flexible hours
+3
Principal Data Scientist
Principal Data Scientist

Yoodli AI Roleplays • Seattle (WA)

Hybrid
USD 180,000 - 320,000
Competitive compensation
Meaningful equity
Hybrid work environment
+1
Principal Data Scientist
Principal Data Scientist

Yoodli, Inc. • Seattle (WA)

On-site
USD 200,000 - 220,000
Competitive compensation
Meaningful equity
Flexible work environment
+1
Head of AI and ML
Head of AI and ML

Xplor Education • United States

On-site
USD 180,000 - 320,000
Senior Fullstack Software Engineer- Analytics
Senior Fullstack Software Engineer- Analytics

Yoodli AI Roleplays • Seattle (WA)

Hybrid
USD 160,000 - 180,000
Equity opportunities
Hybrid work model
Senior Fullstack Software Engineer (Analytics)
Senior Fullstack Software Engineer (Analytics)

Yoodli, Inc. • Seattle (WA)

Hybrid
USD 160,000 - 180,000
Significant equity opportunities
Competitive compensation