Senior Applied Scientist, LLM & Agent Evaluation (Remote)

Ladders

United States

Teletrabalho

USD 200 000 - 300 000

Tempo integral

há 1 hora
Torna-te num dos primeiros candidatos

Recebe uma resposta deste empregador

O teu currículo e carta de apresentação, adaptados exatamente ao que estão a contratar.

Ultrapassa os filtros ATS

Vantagens oferecidas por esta oferta de emprego

Cutting-edge AI tech
Client engagement
Research impact
Advanced experimental tools
Growth & mentorship

Resumo da oferta

Ladders is seeking a Principal Applied Scientist to lead applied research and experimentation across LLM and agent systems. This remote role emphasizes evaluation rigor, production-ready solutions, and collaboration with client stakeholders to define success metrics and experimental design.

You will balance scientific depth with hands-on Python development and problem solving, delivering production-ready research tooling and dashboards for client environments.

Qualificações

  • 5–7 years in LLM/agent evaluation and production readiness.
  • Proven track record with planner–executor architectures.
  • Strong design of experiments and statistical inference.
  • Experience with multi-objective optimization and cost-aware ML.
  • Familiarity with LLM routing and production tooling.

Responsabilidades

  • Lead end-to-end evaluation workflow design for planner–executor agents.
  • Define scenarios and grading specs with client teams.
  • Develop an LLM-based user simulator for realistic multi-turn conversations.
  • Implement parallel experiments with configuration management.
  • Package and deploy optimization levers for initial rollout.
  • Compute core metrics and compare experiments statistically.
  • Build a locally hosted dashboard for insights and tracking.

Conhecimentos

LLM evaluation
Agent evaluation methods
Python development
Experimental design
Statistics
Production tooling
Client engagement

Formação académica

PhD or MS in CS/ML

Ferramentas

Python
Experiment tracking

Descrição da oferta de emprego

Ladders is seeking a Principal Applied Scientist to lead applied research and experimentation across LLM and agent systems. This remote role emphasizes evaluation rigor, production-ready solutions, and collaboration with client stakeholders to define success metrics and experimental design.

You will balance scientific depth with hands-on Python development and problem solving, delivering production-ready research tooling and dashboards for client environments.

Obtém a tua avaliação gratuita e confidencial do currículo.

ou arrasta e larga o ficheiro aqui.

Similar jobs

Ofertas semelhantes que vale a pena comparar

Principal Applied Scientist
Principal Applied Scientist

Ladders • EUA

Teletrabalho
USD 200 000 - 300 000
Cutting-edge AI tech
Client engagement
Research impact
+2
Applied Scientist, LLM Evaluation & Optimization Lead
Applied Scientist, LLM Evaluation & Optimization Lead

robotsandpencils • EUA

Teletrabalho
USD 120 000 - 160 000
Senior AI Platform Engineer – LLM Agents (Remote)
Senior AI Platform Engineer – LLM Agents (Remote)

tldr • EUA

Teletrabalho
USD 250 000 - 300 000
Remote across US/Canada
Biannual team offsites
Flexible PTO
+5
Remote AI Research Engineer — Agent & LLM Systems
Remote AI Research Engineer — Agent & LLM Systems

Softechub • EUA

Teletrabalho
USD 138 000 - 179 000
Staff Applied Scientist
Staff Applied Scientist

robotsandpencils • EUA

Teletrabalho
USD 120 000 - 160 000
Remote Senior Staff Applied Scientist - Science Platform
Remote Senior Staff Applied Scientist - Science Platform

Ladders • EUA

Teletrabalho
USD 233 000 - 326 000
Comprehensive healthcare benefits
401K with employer match
Global benefit programs
+4
Senior AI Engineer: Remote LLM Systems & Guardrails
Senior AI Engineer: Remote LLM Systems & Guardrails

Ladders • EUA

Teletrabalho
USD 180 000 - 225 000
Bonus eligibility
Mentor and lead engineering teams
Work on high-stakes, ambiguous problem
Remote Senior Software Engineer, LLM Evaluation & QA Tools
Remote Senior Software Engineer, LLM Evaluation & QA Tools

OpenTrain AI, Inc. • EUA

Teletrabalho
USD 83 000 - 124 000
Remote AI/ML Engineer: LLM Agents & Backend
Remote AI/ML Engineer: LLM Agents & Backend

24-MAG • EUA

Teletrabalho
USD 220 000 - 300 000
Fully remote
Remote Principal Data Scientist - Measurement & Experiments
Remote Principal Data Scientist - Measurement & Experiments

Ladders • EUA

Teletrabalho
USD 173 000 - 291 000
Remote work flexibility
Equity awards based on performance and
Executive-level exposure
+1