Staff Data Scientist, Agent Improvement Loop

Jobgether; México; Remote; MXN 900,000 - 1,300,000; Full time

Benefits offered by this job

Unlimited PTO

Fully remote

Flexible hours around Central Time

Professional development opportunities

International team

Job summary

Jobgether is seeking a Staff Data Scientist to lead evaluation and optimization of AI-powered agents in a fast-paced, international environment. You will design evaluation strategies, analyze production traces, and translate insights into measurable improvements across product, engineering, and customer success teams.

You will mentor colleagues, contribute to the evolution of agent capabilities, and develop production-quality Python and TypeScript tools, shaping the quality and reliability of

Qualifications

• 8+ years delivering data science solutions with measurable impact.
• Hands-on evaluation of LLM-based systems and AI agents.
• Proficient in Python and SQL; TypeScript willingness.
• Strong software engineering practices and CI/CD experience.
• Experience with production GenAI systems and tools.
• Ability to translate traces into actionable insights.

Responsibilities

• Own and develop the evaluation strategy for AI agents.
• Analyze agent traces and conversations at scale to identify failure patterns.
• Drive improvement cycle with actionable fixes and before/after comparisons.
• Improve agent skills, orchestration, and supporting infrastructure.
• Collaborate with Delivery and Customer Success to enhance workflows.
• Design and validate LLM-as-a-judge evaluation systems.
• Develop production-quality Python and TypeScript code for tools.
• Mentor data scientists and engineers in measurement and evaluation.

Skills

Python

SQL

TypeScript

CI/CD

LangGraph

LangSmith

LLM evaluation

NLP

A/B testing

Education

Bachelor's/Master's in CS/Stats/Math

Tools

LangGraph

LangSmith

Vertex AI

SageMaker

MLflow

Job description

As a Staff Data Scientist, you will play a key role in improving the performance, reliability, and quality of AI-powered agents in a fast-paced, international environment. You will lead the development of evaluation strategies, analyze production data, and transform insights into measurable improvements. Working closely with Product, Engineering, Delivery, and Customer Success teams, you will help identify performance gaps and drive continuous optimization. You will contribute to the evolution of agent capabilities, conversational experiences, and supporting technical infrastructure. This role combines advanced data science, hands‑on engineering, experimentation, and strategic thinking. You will also mentor colleagues and help establish best practices for evaluating and improving AI systems at scale. Your work will directly influence customer outcomes and the development of next-generation conversational AI solutions.

Mexico

Accountabilities:

• Own and develop the evaluation strategy for AI agents, including pre-release behavioral testing, production monitoring, and automated evaluation systems.
• Conduct continuous analysis of agent traces and conversations at scale to identify recurring failure patterns, determine root causes, and prioritize improvement opportunities based on business impact.
• Drive the complete agent improvement cycle by translating analytical findings into actionable fixes, implementing improvements, and measuring their effectiveness through robust before-and-after comparisons and causal analysis.
• Improve agent skills, orchestration frameworks, supporting infrastructure, and developer tools using evidence gathered from production traces, collaborating with relevant engineering teams and contributing directly to their codebases.
• Partner with Delivery and Customer Success teams to improve underperforming customer conversational workflows and establish repeatable processes for maintaining high-quality agent performance.
• Design, calibrate, and validate LLM-as-a-judge evaluation systems against human assessments, measuring consistency, reliability, limitations, and overall effectiveness.
• Develop and maintain production-quality Python and TypeScript code for evaluation and analytical tools, ensuring appropriate testing, documentation, version control, and CI/CD integration.
• Mentor data scientists and engineers in measurement methodologies, experimentation, evaluation practices, and data-driven decision‑making.
• Contribute to quarterly and annual strategic planning, helping define priorities and long‑term initiatives for improving AI agent quality.

Requirements:

• Bachelor's or Master's degree in a technical discipline such as Computer Science, Statistics, Applied Mathematics, Operations Research, Economics, or a related field.
• At least 8 years of experience delivering data science solutions with measurable business impact in cross‑functional environments.
• Demonstrated hands‑on experience evaluating LLM-based systems or AI agents, including evaluation design, LLM-as-a-judge methodologies, evaluation rubrics, and judge calibration.
• Advanced proficiency in Python and SQL, with working knowledge of TypeScript or the willingness to develop expertise in it.
• Strong software engineering practices, including version control, code reviews, automated testing, CI/CD, and experience building and deploying tools for shared production environments.
• Practical experience working with production GenAI systems, including prompt engineering, tool design, agent frameworks such as LangGraph, and LLM observability platforms such as LangSmith.
• Ability to transform complex and unstructured trace and conversational data into actionable insights and communicate findings effectively to both technical and non‑technical stakeholders.
• Strong product and business acumen, with the ability to connect technical quality improvements to customer experience and measurable business outcomes.
• Excellent analytical thinking, problem‑solving, communication, and collaboration skills, combined with the ability to work independently and influence cross‑functional teams.
• Experience with conversational AI, customer‑facing agents, or tool‑using AI systems is considered a strong advantage.
• Knowledge of Natural Language Processing (NLP), clustering techniques, and methodologies for discovering and categorizing recurring system failures is a plus.
• Familiarity with causal inference and experimentation methods, including A/B testing, difference‑in‑differences, and pre/post analysis, as well as the ability to recognize the limitations of available data.
• Exposure to MLOps or LLMOps platforms such as Vertex AI, Amazon SageMaker, MLflow, or similar technologies, as well as big data tools, is desirable.
• Understanding of AI safety, responsible AI principles, and best practices for developing reliable AI systems is an advantage.
• Previous mentoring experience and the ability to support the professional development of other data scientists and engineers are highly valued.
• Adaptability and comfort working in an ambiguous, rapidly evolving environment.

Benefits:

• Competitive compensation aligned with market standards.
• Unlimited paid time off (PTO).
• Fully remote working opportunities, with working hours offering approximately three hours of flexibility around Central Time.
• Flexible working arrangements focused on results, autonomy, and impact.
• Opportunity to work in a dynamic, fast‑growing startup environment.
• International and collaborative team with opportunities to work alongside professionals from different backgrounds and markets.
• Strong opportunities for professional development, career progression, and increased responsibilities.
• Inclusive workplace culture that values diversity, initiative, simplicity, excellence, and individual contributions.
• An environment that encourages innovation, ownership, and the freedom to bring forward new ideas.