Staff Applied Scientist

robotsandpencils; Stati Uniti; Remoto; USD 120.000 - 160.000; Tempo pieno

Descrizione del lavoro

robotsandpencils is seeking a senior role to lead the evaluation framework for production planner-executor agents. You will blend applied research with hands-on engineering to determine model choices and optimization strategies that meet quality goals while reducing cost and latency.

The role spans experiment design, simulation, analysis, and delivering a dashboard to guide rollout decisions for customers. Strong statistical methods and tooling in Python are essential.

Competenze

• Experience evaluating LLMs and multi-step, tool-using agents in production
• Strong experimental design and statistical inference skills
• Experience with cost-aware ML or multi-objective optimization
• Understanding of model routing, inference behavior, and cost controls such as caching

Mansioni

• Build an end-to-end evaluation workflow, including scenario formats, a multi-turn simulated user, isolated experiment runs, grading, and reporting
• Work with customer engineers to define scenario and grading standards, then validate the framework against production interaction logs
• Implement reusable optimization options, such as model cascades or caching, and instrument model calls to capture token use, latency, cache behavior, and cost
• Compare configurations on matched scenarios using confidence intervals and non-inferiority checks; clearly identify inconclusive results
• Translate measured quality and cost differences into production-scale projections and present the cost-quality trade-offs
• Deliver a locally deployable dashboard, run staged experiment campaigns, document integration guidance, and advise stakeholders on go/no-go decisions

Conoscenze

LLM evaluation

Experimental design

Statistical inference

Cost-aware ML

Multi-objective optimization

Model routing

LLM tracing

Python tooling

Strumenti

Python

LLM tracing tools

Descrizione del lavoro

Role overview

Lead the design and delivery of a rigorous evaluation framework for a production planner-executor agent. You will combine applied research with hands-on engineering, helping a customer determine which models and optimization strategies meet quality goals while improving cost and latency. The work spans experiment design, simulation, analysis, and a dashboard the customer can use to guide rollout decisions.

Responsibilities

• Build an end-to-end evaluation workflow, including scenario formats, a multi-turn simulated user, isolated experiment runs, grading, and reporting.
• Work with customer engineers to define scenario and grading standards, then validate the framework against production interaction logs.
• Implement reusable optimization options, such as model cascades or caching, and instrument model calls to capture token use, latency, cache behavior, and cost.
• Compare configurations on matched scenarios using confidence intervals and non-inferiority checks; clearly identify inconclusive results.
• Translate measured quality and cost differences into production-scale projections and present the cost-quality trade-offs.
• Deliver a locally deployable dashboard, run staged experiment campaigns, document integration guidance, and advise stakeholders on go/no-go decisions.

Requirements

• Practical experience evaluating LLMs and multi-step, tool-using agents in production, including state-based verification, trajectory scoring, judging, or user simulation.
• Strong experimental design and statistical inference skills, including paired comparisons, non-inferiority testing, and power analysis.
• Experience with cost-aware ML or multi-objective optimization, including Pareto analysis and estimating business impact from deployment costs.
• Understanding of model routing, inference behavior, and cost controls such as caching, context compaction, and reasoning budgets.
• Experience with LLM tracing and observability, plus the ability to build reliable research tooling, data pipelines, and sandboxed execution in Python.
• Strong customer-facing communication skills: you can scope research work, explain results to technical and business audiences, and lead engineering standards or mentor peers.

Nice to have

• Publications or open-source work in agent evaluation, LLM systems, or efficient inference; experience with client-adoptable dashboards or LLM tracing tools; or work in regulated, high-stakes domains.