Principal Applied Scientist

Jobgether

United States

Remote

USD 200.000 - 300.000

Vollzeit

Vor 2 Tagen
Sei unter den ersten Bewerbenden

Erhalte eine Antwort von diesem Arbeitgeber

Dein Lebenslauf und dein Anschreiben, genau auf das zugeschnitten, was gesucht wird.

Schaffe es an den ATS-Filtern vorbei

Benefits dieser Stelle

Remote work within the United States
Hands-on leadership in AI evaluation
Client-facing, enterprise-focused role

Zusammenfassung

Jobgether, on behalf of a partner company, seeks a Principal Applied Scientist in the United States to lead the development of a rigorous evaluation framework for enterprise AI agents. You will collaborate with engineering teams to assess planner and executor models, and production-quality tools that make evaluation repeatable, from user simulation to statistical insights.

This research-focused, customer-facing role requires building automated insights, leading experimentation campaigns, and

Qualifikationen

  • Experience with LLM and AI agent evaluation methodologies.
  • Hands-on experience building or evaluating production-grade planner-executor systems.
  • Strong expertise in experimental design and statistical inference, including paired testing and confidence intervals.

Aufgaben

  • Design and lead an end-to-end evaluation framework for planner-executor agents.
  • Develop an LLM-powered user simulator and validate its behavior against production logs.
  • Build isolated, parallel experiment execution systems with configurable runs and adapters.
  • Instrument model calls and attribute token usage, latency, and costs to requests.
  • Establish grading methods and report KPIs like verified success rates and p95 latency.
  • Translate experimental results into projected financial impact and business value.
  • Lead staged experimentation campaigns and collaborate with engineers and business stakeholders.

Kenntnisse

LLM evaluation
Statistical inference
Experiment design
Python tooling
Communication with stakeholders
Mentoring engineers

Tools

LangFuse
LangSmith
Python

Jobbeschreibung

This position is listed on behalf of a partner company, which manages all applications and next steps. Our partner is looking for a Principal Applied Scientist based in the United States.

We are seeking an experienced Principal Applied Scientist to lead the development of a rigorous evaluation framework for enterprise AI agents. In this research-intensive, customer-facing role, you will work closely with engineering teams to assess planner models, execution models, and inference optimization strategies through controlled, statistically sound experiments. You will own both the experimental methodology and the production-quality tools that make evaluation repeatable, from realistic user simulation and isolated experiment execution to statistical analysis and automated insights. Your work will help distinguish meaningful improvements from inconclusive results, quantify the relationship between quality and cost, and translate technical findings into measurable business value. You will also provide technical leadership, establish evaluation standards, and deliver actionable recommendations that support enterprise deployment decisions.

Accountabilities
  • Design and lead an end-to-end evaluation framework for planner-executor agents, covering scenario definition, user simulation, configuration, execution, grading, comparison, and insight generation.
  • Define scenario schemas, grading specifications, and validation processes in collaboration with engineering teams, ensuring scenarios capture goals, personas, starting states, and expected outcomes.
  • Develop an LLM-powered user simulator capable of realistic multi-turn conversations, using hidden goals and stop conditions, and validate its behavior against production interaction logs.
  • Build isolated, parallel experiment execution systems with configurable runs and adapters that route model traffic through existing LLM gateways.
  • Develop and implement an initial selection of optimization plug-ins, such as failure-based model cascading, cache-aware model switching, and prompt caching.
  • Instrument model calls and integrate tracing systems to attribute token consumption, cache utilization, latency, and costs to individual requests.
  • Establish grading methods based primarily on verified end states, with optional LLM-based evaluation of answer and planning quality.
  • Calculate and report key performance indicators, including verified success rates, cost per successful task, p95 latency, and failures by pipeline stage.
  • Apply rigorous statistical methods, including paired comparisons, confidence intervals, win/loss/tie analysis, and non-inferiority testing, while clearly identifying inconclusive findings.
  • Translate experimental results into projected financial impact at production scale and develop cost-versus-quality analyses to guide optimization decisions.
  • Build an automated insight engine that uses statistical computation for all reported metrics and LLM-generated explanations to communicate evidence-backed findings.
  • Develop a locally hosted, API-first dashboard with launch, live monitoring, results, comparison, insights, and tracing capabilities, supported by straightforward deployment and integration documentation.
  • Lead staged experimentation campaigns covering production baselines, model screening, optimization testing, and validation of the strongest combinations.
  • Assess how well agent performance generalizes across scenario categories, potentially incorporating data analytics benchmark scenarios.
  • Deliver final recommendations with confidence intervals and projected dollar impact, alongside technical reports, runbooks, and integration guides.
  • Collaborate with engineers and business leaders to establish evaluation criteria, success thresholds, and acceptable quality margins, communicating findings to both technical and non-technical stakeholders.
  • Set experimental standards, provide technical leadership, and mentor engineers in rigorous agent evaluation and AI-assisted development practices.
Requirements
  • Demonstrated experience with LLM and AI agent evaluation methodologies, including state-based verification, LLM-as-judge or agent-as-judge approaches, trajectory-level scoring, and user simulation.
  • Hands-on experience building or evaluating production-grade planner-executor systems or other multi-step, tool-using AI agent architectures.
  • Strong expertise in experimental design and statistical inference, including paired testing, non-inferiority analysis, power analysis, confidence intervals, and the ability to recognize when evidence is inconclusive.
  • Experience with multi-objective optimization or cost-aware machine learning, including Pareto frontier analysis and translating technical results into projected financial savings.
  • Familiarity with LLM routing, model cascades, model selection, and inference optimization techniques such as prompt and plan caching, context compaction, reasoning budgets, and cache-aware model switching.
  • Understanding of LLM inference internals and the ability to evaluate the impact of optimization strategies on quality, latency, and cost.
  • Experience implementing observability and tracing for LLM systems, including request-level attribution of tokens, cache usage, latency, and cost.
  • Ability to develop production-quality research tooling using Python services, containerized and sandboxed execution environments, and data pipelines.
  • Previous customer-facing or consulting experience, including scoping and delivering research-oriented technical engagements.
  • Excellent communication skills, with the ability to explain complex technical findings, respond confidently to detailed questions, and connect results to business outcomes.
  • Additional advantages include:
    • Publications or open-source contributions in LLM evaluation, AI agents, or efficient inference.
    • Experience developing internal dashboards or web applications that can be adopted and maintained by client engineering teams.
    • Practical experience with LLM gateways and tracing platforms such as LangFuse or LangSmith.
    • Background working in regulated or high-stakes enterprise environments where evaluation rigor is essential.
    • A demonstrated understanding of real-world cost-quality trade-offs gained through production experience.
Benefits
  • Annual salary range of $200,000–$300,000.
  • Remote work opportunity within the United States.
  • Customer-facing, research-driven work with substantial ownership of technical strategy, experimentation, and implementation.
  • Opportunity to shape enterprise AI evaluation practices and deliver tools that support real-world deployment decisions.
  • Close collaboration with engineering teams and business stakeholders.
  • Significant technical leadership and mentoring responsibilities.
  • Opportunity to work on advanced AI agent architectures, inference optimization, and measurable enterprise AI performance improvements.
  • The role involves approximately 20% travel within the United States, including onsite collaboration and client-related activities.
  • Willingness to attend an onsite final interview and participate in onsite onboarding is expected.
Hol dir deinen kostenlosen, vertraulichen Lebenslauf-Check.

oder ziehe deine Datei hierhin.

Similar jobs

Ähnliche Jobs, die dir auch gefallen könnten

Principal Applied Scientist
Principal Applied Scientist

Robots & Pencils • USA

Remote
USD 200.000 - 300.000
Staff Applied Scientist
Staff Applied Scientist

robotsandpencils • USA

Remote
USD 120.000 - 160.000
Senior Applied AI Engineer
Senior Applied AI Engineer

People in AI • New York (NY)

Vor Ort
USD 225.000 - 275.000
Director, Applied AI
Director, Applied AI

Zoominfo • USA

Hybrid
USD 233.000 - 366.000
Hybrid work arrangement
Comprehensive benefits program
Senior Agentic AI Engineer
Senior Agentic AI Engineer

Select Minds LLC • Dallas (TX)

Hybrid
USD 180.000 - 240.000
Hybrid work arrangement
Competitive salary
Opportunity for advancement
Member of Technical Staff – Applied ML
Member of Technical Staff – Applied ML

Raydar • New York (NY)

Vor Ort
USD 175.000 - 350.000
Equity
Visa sponsorship
Member of Technical Staff, Cortex Lab (applied AI research)
Member of Technical Staff, Cortex Lab (applied AI research)

Confidential • USA

Remote
USD 200.000 - 250.000
Equity compensation
Performance-based bonuses
Health insurance reimbursement
+2
AI Engineer
AI Engineer

Moring AI • Raleigh (NC)

Vor Ort
USD 60.000 - 80.000
ML Research Engineer
ML Research Engineer

Radical Ai, Inc. • New York (NY)

Vor Ort
USD 235.000 - 295.000
AI Agents Applied Research/Engineering Lead - Vice President
AI Agents Applied Research/Engineering Lead - Vice President

Testing Site - JPMC • New York (NY)

Vor Ort
USD 180.000 - 280.000