AI Agent Quality & Evaluation Scientist

Clera; Palo Alto (CA); On-site; USD 150,000 - 210,000; Full time

Job summary

Clera in Palo Alto, CA seeks a data-science leader to own end-to-end evaluation for autonomous AI agents across email, calendar, and business software tasks. You will transform ambiguous product behavior into rigorous, actionable evaluation criteria guiding engineering and product decisions.

The role requires 5+ years in data science/ML, strong Python/SQL, and expertise in evaluation frameworks, experimental design, and ground-truth data development.

Qualifications

• 5+ years in data science, machine learning, or analytics roles focusing on evaluation systems for production
• Experience designing evaluation frameworks, grading systems, and success criteria for ML/AI in production
• Strong Python and SQL proficiency with automated data pipelines and production-quality analysis code
• Statistical design knowledge: sampling, variance, uncertainty quantification, bias detection
• Ground-truth data development: labeling guidelines, annotation quality control, ambiguity resolution
• Knowledge of LLM behavior, tool use, retrieval systems, multi-step execution, and practical failure modes of language model systems
• Ability to connect quantitative patterns to traces and identify failure origins across model, prompt, data, and logic
• Experience communicating evaluation results and trade-offs to technical and non-technical stakeholders
• Comfort operating with high ownership in ambiguous, fast-moving environments
• Experience with LLM-as-a-judge or benchmarking platforms is a strong plus

Responsibilities

• Architect and maintain automated evaluation pipelines measuring agent quality across capabilities and product surfaces
• Translate agent capabilities into explicit success criteria, including pass, partial-pass, and failure definitions
• Build representative gold datasets and regression suites covering workflows, edge cases, ambiguous requests, and adversarial scenarios
• Define and track metrics such as task success, tool-selection accuracy, instruction adherence, factual consistency, latency, cost, and reliability
• Design deterministic and model-based graders, calibrate LLM-as-a-judge systems, and measure grader agreement, false positives, and false negatives
• Analyze traces, tool calls, model outputs, and production outcomes to identify root causes and build a taxonomy
• Compare models, prompts, tools, and capability implementations using offline experiments and production evidence
• Build dashboards and release-quality signals that make evaluation results understandable and actionable
• Partner with capability engineers to recommend improvements and verify fixes raise quality without regressions

Skills

Evaluation systems

Metrics frameworks

Quality measurement

Production ML

Statistical design

Experiment design

Ground-truth data

LLM behavior understanding

Stakeholder communication

Ownership in ambiguity

LLM-as-a-judge knowledge

Tools

Python

SQL

Job description

Clera in Palo Alto, CA seeks a data-science leader to own end-to-end evaluation for autonomous AI agents across email, calendar, and business software tasks. You will transform ambiguous product behavior into rigorous, actionable evaluation criteria guiding engineering and product decisions.

The role requires 5+ years in data science/ML, strong Python/SQL, and expertise in evaluation frameworks, experimental design, and ground-truth data development.