Staff Data Scientist — AI Agent Quality & Improvement Loop

yalochatinc

Ciudad de México

Híbrido

USD 140.000 - 230.000

Jornada completa

Hace 11 días

Una candidatura completa en un minuto

Currículum y carta de presentación personalizados, listos para enviar.

Supera los filtros ATS

Ventajas ofrecidas por este puesto de trabajo

Unlimited PTO
Remote work available
Flexible time
Startup environment
International teamwork
Career growth opportunities

Descripción de la vacante

Yalo is hiring a Staff Data Scientist to improve our AI agents. You will own how we evaluate agents, run large-scale error analysis, and turn findings into fixes across skills, prompts, and releases.

You will collaborate with product, engineering, delivery and customer success to move real customer flows from underperforming to reliably healthy. You will write production-grade Python and TypeScript, mentor teams, and help shape quarterly plans for agent quality.

Formación

  • BS or MS in a technical field (CS, Statistics, OR, Economics, etc.).
  • 8+ years delivering data science solutions with measurable business impact.
  • Hands‑on experience evaluating LLM‑based systems or agents: eval design, LLM-as-judge, rubric design, and judge calibration.
  • Expert in Python and SQL. Comfortable in TypeScript, or ready to be.
  • Strong production engineering habits: version control, code review, testing, CI/CD, and shipping tools others use.
  • Experience with GenAI systems in production: prompt and tool design, agent frameworks, and LLM observability tooling.
  • Skilled at turning messy trace and conversation data into clear findings for technical and non-technical audiences.
  • Strong product and business sense. You frame quality work in terms of customer outcomes.

Responsabilidades

  • Own the evals strategy for our agents: pre-release behavioral testing, production monitoring, and the judges that score conversations.
  • Run continuous error analysis on agent traces at scale. Find failure patterns, attribute them to skills, prompts or releases, and rank improvement opportunities by impact.
  • Close the loop. Turn findings into fixes and new evals, ship them, and measure the result with sound before-and-after and causal methods.
  • Improve our agent skills, harness and CLI using trace evidence. Work through the owning teams' repos and review process.
  • Partner with delivery and customer teams to take real customer flows from underperforming to reliably healthy, and make the process repeatable for others.
  • Calibrate and validate LLM judges against human review. Measure their stability, report their limits, and set the bar for what counts as evidence at Yalo.
  • Write production-quality Python and TypeScript for evals and analysis tooling, with tests and CI, so it can live in a shared codebase.
  • Mentor data scientists and engineers on measurement, experimentation and evaluation practice.
  • Help shape quarterly and annual planning for agent quality.

Conocimientos

Python
SQL
TypeScript
LLM evaluation
Data analysis
Experimentation
Communication

Educación

BS or MS in Computer Science or related field

Herramientas

LangGraph
LangSmith
CI/CD tooling

Descripción del empleo

Yalo is hiring a Staff Data Scientist to improve our AI agents. You will own how we evaluate agents, run large-scale error analysis, and turn findings into fixes across skills, prompts, and releases.

You will collaborate with product, engineering, delivery and customer success to move real customer flows from underperforming to reliably healthy. You will write production-grade Python and TypeScript, mentor teams, and help shape quarterly plans for agent quality.

Consigue la evaluación confidencial y gratuita de tu currículum.

o arrastra y suelta tu archivo aquí

Similar jobs

Puestos de trabajo similares que vale la pena comparar

Staff Data Scientist: Agent Improvement & Quality Lead
Staff Data Scientist: Agent Improvement & Quality Lead

Yalo • EE. UU.

A distancia
USD 180.000 - 260.000
Unlimited PTO
Remote work
Flexible hours
+3
Staff Data Scientist: Agent Improvement & Evaluation
Staff Data Scientist: Agent Improvement & Evaluation

Yalo Inc. • EE. UU.

A distancia
USD 180.000 - 240.000
Unlimited PTO
Competitive rewards
Remote work
+3
Staff Data Scientist, Agent Improvement Loop
Staff Data Scientist, Agent Improvement Loop

Yalo • EE. UU.

A distancia
USD 180.000 - 260.000
Unlimited PTO
Remote work
Flexible hours
+3
Staff Data Scientist, Agent Improvement Loop
Staff Data Scientist, Agent Improvement Loop

Yalo Inc. • EE. UU.

A distancia
USD 180.000 - 240.000
Unlimited PTO
Competitive rewards
Remote work
+3
Senior AI Engineer — Quality & Agent Systems (Hybrid)
Senior AI Engineer — Quality & Agent Systems (Hybrid)

Glean • Mountain View (CA)

Híbrido
USD 200.000 - 280.000
Home office stipend
Education stipend
Wellness stipend
+1
Member of Technical Staff, Agents
Member of Technical Staff, Agents

Clera • Palo Alto (CA)

Presencial
USD 200.000 - 300.000
Member of Technical Staff, Agents
Member of Technical Staff, Agents

Clera • EE. UU.

Presencial
USD 200.000 - 300.000
Staff Data Scientist: AI System Quality & Experiments
Staff Data Scientist: AI System Quality & Experiments

Syllo • EE. UU.

A distancia
USD 120.000 - 165.000
Member of Technical Staff, Agents
Member of Technical Staff, Agents

Royper Royco LLC • Palo Alto (CA)

Presencial
USD 200.000 - 300.000
AI Engineer — Build Production Autonomous Agents
AI Engineer — Build Production Autonomous Agents

Yolk • Illinois

Híbrido
USD 120.000 - 150.000
Equity
Mentorship