Senior AI Evaluation Scientist (Planner-Executor)

Robots & Pencils; Stati Uniti; Remoto; USD 200.000 - 300.000; Tempo pieno

Descrizione del lavoro

Robots & Pencils is seeking a Principal Applied Scientist to lead an evaluation framework for a planner-based agent. You will benchmark agents across models, executors, and optimization levers, delivering measurable dollars at production volume.

This research-heavy, customer-facing role owns both method and tooling, including a scenario-driven simulator, isolated experiments, and automated insights in a locally hosted dashboard. You’ll drive actionable recommendations.

Competenze

• Experience with LLM and agent evaluation methods.
• Experience building or evaluating planner-executor architectures in production.
• Strong background in experimental design and statistics.
• Experience with cost-aware ML and Pareto frontier analysis.
• Familiarity with LLM routing and cost levers.
• Ability to build production-quality research tooling.
• Customer-facing consulting experience.
• Ability to explain findings to non-technical audiences.

Mansioni

• Design end-to-end evaluation workflow for planner-executor agent.
• Define scenario schema, grading spec and validators with client team.
• Build an LLM-based user simulator with realistic multi-turn chats.
• Implement isolated, parallel experiment execution and a harness.
• Package optimization levers as plug-ins and select top 3-4.
• Tag model calls and send traces to a trace store.

Conoscenze

LLM evaluation

planner-executor

experimental design

statistical inference

cost-aware ML

observability

production tooling

client-facing

Strumenti

LangFuse

LangSmith

Descrizione del lavoro

Robots & Pencils is seeking a Principal Applied Scientist to lead an evaluation framework for a planner-based agent. You will benchmark agents across models, executors, and optimization levers, delivering measurable dollars at production volume.

This research-heavy, customer-facing role owns both method and tooling, including a scenario-driven simulator, isolated experiments, and automated insights in a locally hosted dashboard. You’ll drive actionable recommendations.