Applied Scientist, LLM Evaluation & Optimization Lead

robotsandpencils; Stati Uniti; Remoto; USD 120.000 - 160.000; Tempo pieno

Descrizione del lavoro

robotsandpencils is seeking a senior role to lead the evaluation framework for production planner-executor agents. You will blend applied research with hands-on engineering to determine model choices and optimization strategies that meet quality goals while reducing cost and latency.

The role spans experiment design, simulation, analysis, and delivering a dashboard to guide rollout decisions for customers. Strong statistical methods and tooling in Python are essential.

Competenze

• Experience evaluating LLMs and multi-step, tool-using agents in production
• Strong experimental design and statistical inference skills
• Experience with cost-aware ML or multi-objective optimization
• Understanding of model routing, inference behavior, and cost controls such as caching

Mansioni

• Build an end-to-end evaluation workflow, including scenario formats, a multi-turn simulated user, isolated experiment runs, grading, and reporting
• Work with customer engineers to define scenario and grading standards, then validate the framework against production interaction logs
• Implement reusable optimization options, such as model cascades or caching, and instrument model calls to capture token use, latency, cache behavior, and cost
• Compare configurations on matched scenarios using confidence intervals and non-inferiority checks; clearly identify inconclusive results
• Translate measured quality and cost differences into production-scale projections and present the cost-quality trade-offs
• Deliver a locally deployable dashboard, run staged experiment campaigns, document integration guidance, and advise stakeholders on go/no-go decisions

Conoscenze

LLM evaluation

Experimental design

Statistical inference

Cost-aware ML

Multi-objective optimization

Model routing

LLM tracing

Python tooling

Strumenti

Python

LLM tracing tools

Descrizione del lavoro

robotsandpencils is seeking a senior role to lead the evaluation framework for production planner-executor agents. You will blend applied research with hands-on engineering to determine model choices and optimization strategies that meet quality goals while reducing cost and latency.

The role spans experiment design, simulation, analysis, and delivering a dashboard to guide rollout decisions for customers. Strong statistical methods and tooling in Python are essential.