Data Scientist, AI Agent Quality & Evaluation

Clera; United States; Remote; USD 120,000 - 190,000; Full time

Job summary

Clera is building an AI executive assistant operating across email, calendars, meetings, and business software. We seek a Data Scientist — Agent Evaluations & Quality to own the measurement system that determines real-world improvements in ambiguous environments.

You will partner with AI Agent Capabilities engineers to generate evidence shaping product decisions and release quality. This high-ownership role sits at the intersection of applied data science, LLM evaluation, and product quality,

Qualifications

• 4+ years in Applied Data Science or Machine Learning roles, with a track record of building and delivering evaluation systems, automated data pipelines, or production ML infrastructure.
• Experience designing automated evaluation frameworks, success criteria, and regression suites for complex AI/ML or agentic systems.
• Production-grade proficiency in Python and SQL, with experience building and maintaining automated analytical pipelines on large datasets.
• Applied statistical and experimental skills: significance testing, variance analysis, and sampling to evaluate non-deterministic AI/ML systems.
• Experience developing labeled datasets, annotation guidelines, and quality-control processes for ground-truth data in dynamic product environments.
• Solid understanding of LLM agent behaviors: tool use, multi-step execution, retrieval, and practical failure modes.
• Demonstrated ability to analyze model traces, tool calls, and outputs to identify root causes across model, prompt, tool, and data layers.
• Experience using production telemetry and observability data to monitor system quality, build dashboards, and analyze real-world user outcomes.

Responsibilities

• Architect and maintain automated evaluation pipelines that measure agent quality across product surfaces.
• Translate agent capabilities into explicit pass, partial-pass, and failure criteria for complex multi-step tasks.
• Build representative gold datasets and regression suites covering real workflows, edge cases, and adversarial scenarios.
• Define meaningful metrics — task success, tool-selection accuracy, instruction adherence, factual consistency, latency, cost, and reliability.
• Design deterministic and model-based graders, calibrate LLM-as-a-judge systems, and track grader agreement.
• Compare models, prompts, and implementations using rigorous offline experiments and production evidence.
• Analyze traces and production outcomes to identify root causes and build a practical failure taxonomy.
• Turn production failures into regression cases and continuously close gaps in evaluation coverage.
• Build dashboards and release-quality signals that make results actionable for engineering, product, and leadership.
• Recommend improvements to capability engineers and verify that fixes raise quality without unacceptable regressions.

Skills

Python

SQL

Evaluation pipelines

Experiment design

Statistical analysis

LLM evaluation

Job description

Clera is building an AI executive assistant operating across email, calendars, meetings, and business software. We seek a Data Scientist — Agent Evaluations & Quality to own the measurement system that determines real-world improvements in ambiguous environments.

You will partner with AI Agent Capabilities engineers to generate evidence shaping product decisions and release quality. This high-ownership role sits at the intersection of applied data science, LLM evaluation, and product quality,