Data Scientist, Agent Evaluations & Quality

Clera; Palo Alto (CA); On-site; USD 150,000 - 210,000; Full time

Job summary

Clera in Palo Alto, CA seeks a data-science leader to own end-to-end evaluation for autonomous AI agents across email, calendar, and business software tasks. You will transform ambiguous product behavior into rigorous, actionable evaluation criteria guiding engineering and product decisions.

The role requires 5+ years in data science/ML, strong Python/SQL, and expertise in evaluation frameworks, experimental design, and ground-truth data development.

Qualifications

• 5+ years in data science, machine learning, or analytics roles focusing on evaluation systems for production
• Experience designing evaluation frameworks, grading systems, and success criteria for ML/AI in production
• Strong Python and SQL proficiency with automated data pipelines and production-quality analysis code
• Statistical design knowledge: sampling, variance, uncertainty quantification, bias detection
• Ground-truth data development: labeling guidelines, annotation quality control, ambiguity resolution
• Knowledge of LLM behavior, tool use, retrieval systems, multi-step execution, and practical failure modes of language model systems
• Ability to connect quantitative patterns to traces and identify failure origins across model, prompt, data, and logic
• Experience communicating evaluation results and trade-offs to technical and non-technical stakeholders
• Comfort operating with high ownership in ambiguous, fast-moving environments
• Experience with LLM-as-a-judge or benchmarking platforms is a strong plus

Responsibilities

• Architect and maintain automated evaluation pipelines measuring agent quality across capabilities and product surfaces
• Translate agent capabilities into explicit success criteria, including pass, partial-pass, and failure definitions
• Build representative gold datasets and regression suites covering workflows, edge cases, ambiguous requests, and adversarial scenarios
• Define and track metrics such as task success, tool-selection accuracy, instruction adherence, factual consistency, latency, cost, and reliability
• Design deterministic and model-based graders, calibrate LLM-as-a-judge systems, and measure grader agreement, false positives, and false negatives
• Analyze traces, tool calls, model outputs, and production outcomes to identify root causes and build a taxonomy
• Compare models, prompts, tools, and capability implementations using offline experiments and production evidence
• Build dashboards and release-quality signals that make evaluation results understandable and actionable
• Partner with capability engineers to recommend improvements and verify fixes raise quality without regressions

Skills

Evaluation systems

Metrics frameworks

Quality measurement

Production ML

Statistical design

Experiment design

Ground-truth data

LLM behavior understanding

Stakeholder communication

Ownership in ambiguity

LLM-as-a-judge knowledge

Tools

Python

SQL

Job description

About The Role

This role sits at the intersection of applied data science and AI product quality for a small, fast-moving AI productivity startup building autonomous agents that handle email, calendar, browser, and business software tasks. You will own the measurement of agent quality end-to-end: turning ambiguous product behavior into rigorous, actionable evaluation systems that directly guide engineering and product decisions.

What You’ll Do

• Architect and maintain automated evaluation pipelines that measure agent quality across capabilities and product surfaces.
• Translate agent capabilities into explicit success criteria, including pass, partial-pass, and failure definitions for complex multi-step tasks.
• Build representative gold datasets and regression suites covering common workflows, edge cases, ambiguous requests, and adversarial scenarios.
• Define and track metrics such as task success, tool-selection accuracy, instruction adherence, factual consistency, latency, cost, and reliability.
• Design deterministic and model-based graders, calibrate LLM-as-a-judge systems, and measure grader agreement, false positives, and false negatives.
• Analyze traces, tool calls, model outputs, and production outcomes to identify root causes and build a useful failure taxonomy.
• Compare models, prompts, tools, and capability implementations using rigorous offline experiments and production evidence.
• Build dashboards and release-quality signals that make evaluation results understandable and actionable for engineering, product, and leadership.
• Partner with capability engineers to recommend improvements and verify that fixes raise quality without unacceptable regressions in cost, latency, or reliability.

What We’re Looking For

• 5+ years in data science, machine learning, or analytics roles, with a focus on evaluation systems, metrics frameworks, or quality measurement for production systems.
• Demonstrated experience designing and implementing evaluation frameworks, grading systems, and success criteria for ML or AI systems in production.
• Strong Python and SQL proficiency with the ability to build automated data pipelines and production-quality analysis code at scale.
• Solid statistical and experimental design knowledge: sampling, variance, uncertainty quantification, bias detection, confounding variables, and significance testing for non-deterministic systems.
• Experience with ground-truth data development: labeling guideline design, annotation quality control, ambiguity resolution, and dataset maintenance.
• Working knowledge of LLM behavior, tool use, retrieval systems, multi-step execution, and practical failure modes of language model systems.
• Ability to connect quantitative patterns to individual system traces and identify failure origins across model, prompt, context, tools, data, and application logic.
• Experience communicating evaluation results, methodology, uncertainty, and trade-offs to both technical and non-technical stakeholders.
• Comfort operating with high ownership in ambiguous, fast-moving environments, independently turning open-ended quality questions into evaluation systems.
• Experience with LLM-as-a-judge systems, agentic or multi-step task evaluation, or benchmarking platforms for AI systems is a strong plus.

Location

On-site in Palo Alto, California, United States. Visa sponsorship is not available for this role.