Staff Data Scientist, Agent Improvement Loop

Jobgether; Brasil; Remote; BRL 320,000 - 520,000; Full time

Benefits offered by this job

Unlimited PTO

Fully remote

Flexible hours around Central Time

Professional development

Job summary

Jobgether is seeking a Staff Data Scientist focused on Agent Improvement Loop in Brazil. You will lead evaluation strategies for AI-powered agents, analyze production traces, and translate insights into concrete improvements across product, engineering, and customer success teams.

You will design scalable evaluation systems, calibrate LLM-based judgments, mentor teammates, and help shape AI system reliability and performance at scale in a fast-growing startup.

Qualifications

• Bachelor’s or Master’s degree in a technical field (CS/Statistics/Math/Operations Research).
• 8+ years of data science experience with measurable business impact.
• Hands-on experience evaluating LLM-based systems or AI agents (design, rubrics, calibration).
• Strong Python and SQL; working knowledge of TypeScript or willingness to learn.

Responsibilities

• Own and develop evaluation strategies for AI agents and pre-release testing.
• Analyze agent traces and conversations at scale to identify failure patterns and root causes.
• Lead improvements across agent skills, orchestration, and tooling with production data.
• Calibrate and validate LLM-as-a-judge systems against human assessments.
• Develop production-quality Python/TypeScript code for evaluation tools and CI/CD.
• Mentor data scientists and engineers in measurement methodologies and experiments.
• Contribute to strategic planning for AI agent quality improvements.

Skills

Python

SQL

LLM evaluation

Experimentation

Data analysis

Communication

Education

Bachelor’s or Master’s in CS/Statistics/Math/OR

Tools

LangGraph

LangSmith

Vertex AI

SageMaker

MLflow

Job description

This position is listed on behalf of a partner company, who manages all applications and next steps. Our partner is looking for a Staff Data Scientist, Agent Improvement Loop based in Brazil.

As a Staff Data Scientist, you will play a key role in improving the performance, reliability, and quality of AI-powered agents in a fast-paced, international environment. You will lead the development of evaluation strategies, analyze production data, and transform insights into measurable improvements. Working closely with Product, Engineering, Delivery, and Customer Success teams, you will help identify performance gaps and drive continuous optimization. You will contribute to the evolution of agent capabilities, conversational experiences, and supporting technical infrastructure. This role combines advanced data science, hands-on engineering, experimentation, and strategic thinking. You will also mentor colleagues and help establish best practices for evaluating and improving AI systems at scale. Your work will directly influence customer outcomes and the development of next-generation conversational AI solutions.

Accountabilities

• Own and develop the evaluation strategy for AI agents, including pre-release behavioral testing, production monitoring, and automated evaluation systems.
• Conduct continuous analysis of agent traces and conversations at scale to identify recurring failure patterns, determine root causes, and prioritize improvement opportunities based on business impact.
• Drive the complete agent improvement cycle by translating analytical findings into actionable fixes, implementing improvements, and measuring their effectiveness through robust before-and-after comparisons and causal analysis.
• Improve agent skills, orchestration frameworks, supporting infrastructure, and developer tools using evidence gathered from production traces, collaborating with relevant engineering teams and contributing directly to their codebases.
• Partner with Delivery and Customer Success teams to improve underperforming customer conversational workflows and establish repeatable processes for maintaining high-quality agent performance.
• Design, calibrate, and validate LLM-as-a-judge evaluation systems against human assessments, measuring consistency, reliability, limitations, and overall effectiveness.
• Develop and maintain production-quality Python and TypeScript code for evaluation and analytical tools, ensuring appropriate testing, documentation, version control, and CI/CD integration.
• Mentor data scientists and engineers in measurement methodologies, experimentation, evaluation practices, and data-driven decision-making.
• Contribute to quarterly and annual strategic planning, helping define priorities and long-term initiatives for improving AI agent quality.

Requirements

• Bachelor's or Master's degree in a technical discipline such as Computer Science, Statistics, Applied Mathematics, Operations Research, Economics, or a related field.
• At least 8 years of experience delivering data science solutions with measurable business impact in cross-functional environments.
• Demonstrated hands-on experience evaluating LLM-based systems or AI agents, including evaluation design, LLM-as-a-judge methodologies, evaluation rubrics, and judge calibration.
• Advanced proficiency in Python and SQL, with working knowledge of TypeScript or the willingness to develop expertise in it.
• Strong software engineering practices, including version control, code reviews, automated testing, CI/CD, and experience building and deploying tools for shared production environments.
• Practical experience working with production GenAI systems, including prompt engineering, tool design, agent frameworks such as LangGraph, and LLM observability platforms such as LangSmith.
• Ability to transform complex and unstructured trace and conversational data into actionable insights and communicate findings effectively to both technical and non-technical stakeholders.
• Strong product and business acumen, with the ability to connect technical quality improvements to customer experience and measurable business outcomes.
• Excellent analytical thinking, problem-solving, communication, and collaboration skills, combined with the ability to work independently and influence cross-functional teams.
• Experience with conversational AI, customer-facing agents, or tool-using AI systems is considered a strong advantage.
• Knowledge of Natural Language Processing (NLP), clustering techniques, and methodologies for discovering and categorizing recurring system failures is a plus.
• Familiarity with causal inference and experimentation methods, including A/B testing, difference-in-differences, and pre/post analysis, as well as the ability to recognize the limitations of available data.
• Exposure to MLOps or LLMOps platforms such as Vertex AI, Amazon SageMaker, MLflow, or similar technologies, as well as big data tools, is desirable.
• Understanding of AI safety, responsible AI principles, and best practices for developing reliable AI systems is an advantage.
• Previous mentoring experience and the ability to support the professional development of other data scientists and engineers are highly valued.
• Adaptability and comfort working in an ambiguous, rapidly evolving environment.

Benefits

• Competitive compensation aligned with market standards.
• Unlimited paid time off (PTO).
• Fully remote working opportunities, with working hours offering approximately three hours of flexibility around Central Time.
• Flexible working arrangements focused on results, autonomy, and impact.
• Opportunity to work in a dynamic, fast-growing startup environment.
• International and collaborative team with opportunities to work alongside professionals from different backgrounds and markets.
• Strong opportunities for professional development, career progression, and increased responsibilities.
• Inclusive workplace culture that values diversity, initiative, simplicity, excellence, and individual contributions.
• An environment that encourages innovation, ownership, and the freedom to bring forward new ideas.

How Jobgether works:

We use an AI-powered matching process to ensure your application is reviewed quickly, objectively, and fairly against the role's core requirements. Our system identifies the top-fitting candidates, and this shortlist is then shared directly with the hiring company. The final decision and next steps (interviews, assessments) are managed by their internal team.

We appreciate your interest and wish you the best!