Principal Applied Scientist

Ladders

United States

Remote

USD 200,000 - 300,000

Full time

Just now
Be an early applicant

Get a reply from this employer

Your resume and cover letter, tailored to exactly what they’re hiring for.

Get past ATS filters

Benefits offered by this job

Cutting-edge AI tech
Client engagement
Research impact
Advanced experimental tools
Growth & mentorship

Job summary

Ladders is seeking a Principal Applied Scientist to lead applied research and experimentation across LLM and agent systems. This remote role emphasizes evaluation rigor, production-ready solutions, and collaboration with client stakeholders to define success metrics and experimental design.

You will balance scientific depth with hands-on Python development and problem solving, delivering production-ready research tooling and dashboards for client environments.

Qualifications

  • 5–7 years in LLM/agent evaluation and production readiness.
  • Proven track record with planner–executor architectures.
  • Strong design of experiments and statistical inference.
  • Experience with multi-objective optimization and cost-aware ML.
  • Familiarity with LLM routing and production tooling.

Responsibilities

  • Lead end-to-end evaluation workflow design for planner–executor agents.
  • Define scenarios and grading specs with client teams.
  • Develop an LLM-based user simulator for realistic multi-turn conversations.
  • Implement parallel experiments with configuration management.
  • Package and deploy optimization levers for initial rollout.
  • Compute core metrics and compare experiments statistically.
  • Build a locally hosted dashboard for insights and tracking.

Skills

LLM evaluation
Agent evaluation methods
Python development
Experimental design
Statistics
Production tooling
Client engagement

Education

PhD or MS in CS/ML

Tools

Python
Experiment tracking

Job description

For our client, we are seeking a Principal Applied Scientist to join the team of a leader in the Information Technology space. This role will drive applied research and experimentation across LLM and agent systems, with a strong focus on evaluation rigor, optimization, and practical deployment in client-facing environments. You will partner closely with customer stakeholders to define success metrics, shape experimental design, and turn research insights into production-ready solutions. The work blends scientific depth with hands‑on implementation, requiring a balance of statistical thinking, Python development, and collaborative problem‑solving.

Location: Remote

Compensation: $200,000 – $300,000 annually

Responsibilities
  • Lead the design of an end‑to‑end evaluation workflow for planner‑executor agents
  • Define scenario schemas and grading specifications in collaboration with the client team
  • Develop an LLM‑based user simulator for realistic multi‑turn conversations
  • Implement isolated, parallel experiment execution with configuration management
  • Package and implement optimization levers for initial deployment
  • Compute core metrics and compare experiments using statistical methods
  • Build a locally hosted dashboard for insights and experiment tracking
Qualifications
  • 5-7 years of experience in LLM and agent evaluation methods
  • Proven track record in building or evaluating planner‑executor architectures in production
  • Strong background in experimental design and statistical inference
  • Experience with multi‑objective optimization and cost‑aware ML
  • Familiarity with LLM routing and inference internals
  • Ability to build production‑quality research tooling in Python
  • Prior customer‑facing or consulting experience
Benefits
  • Opportunity to work on cutting‑edge AI technologies
  • Collaborative environment with direct client engagement
  • Chance to influence significant business outcomes through research
  • Access to advanced experimental tools and methodologies
  • Potential for professional growth and mentorship opportunities

Our client is an equal opportunity employer. We encourage you to apply even if you don’t meet every qualification—your background could be exactly what this team needs.

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Principal Applied Scientist
Principal Applied Scientist

Jobgether • United States

Remote
USD 200,000 - 300,000
Remote work within the United States
Hands-on leadership in AI evaluation
Client-facing, enterprise-focused role
Principal Applied Scientist
Principal Applied Scientist

Robots & Pencils • United States

Remote
USD 200,000 - 300,000
Staff Applied Scientist
Staff Applied Scientist

robotsandpencils • United States

Remote
USD 120,000 - 160,000
Senior Applied Scientist, LLM & Agent Evaluation (Remote)
Senior Applied Scientist, LLM & Agent Evaluation (Remote)

Ladders • United States

Remote
USD 200,000 - 300,000
Cutting-edge AI tech
Client engagement
Research impact
+2
Member of Technical Staff, Cortex Lab (applied AI research)
Member of Technical Staff, Cortex Lab (applied AI research)

Confidential • United States

Remote
USD 200,000 - 250,000
Equity compensation
Performance-based bonuses
Health insurance reimbursement
+2
AI Agents Applied Engineer - Senior Associate
AI Agents Applied Engineer - Senior Associate

Testing Site - JPMC • New York (NY)

On-site
USD 160,000 - 240,000
ML Research Engineer Brooklyn, NY On-site $235,000 – $295,000 · per year · USD
ML Research Engineer Brooklyn, NY On-site $235,000 – $295,000 · per year · USD

Nodi • New York (NY)

On-site
USD 235,000 - 295,000
AI Research Engineer | Remote
AI Research Engineer | Remote

Crossing Hurdles • United States

Remote
USD 300,000 - 700,000
Applied AI Engineer
Applied AI Engineer

AItoolnavio • United States

Remote
USD 140,000 - 190,000
Fully remote
Long-term role
Exposure to advanced LLM applications
AI Agents Applied Research/Engineering Lead - Vice President
AI Agents Applied Research/Engineering Lead - Vice President

JPMorgan Chase & Co. • New York (NY)

On-site
USD 180,000 - 240,000