Principal Enterprise AI Evaluation Scientist (Remote)

Jobgether

United States

Remote

USD 200,000 - 300,000

Full time

2 days ago
Be an early applicant

A complete application in a minute

Tailored resume and cover letter, ready to send.

Get past ATS filters

Benefits offered by this job

Remote work within the United States
Hands-on leadership in AI evaluation
Client-facing, enterprise-focused role

Job summary

Jobgether, on behalf of a partner company, seeks a Principal Applied Scientist in the United States to lead the development of a rigorous evaluation framework for enterprise AI agents. You will collaborate with engineering teams to assess planner and executor models, and production-quality tools that make evaluation repeatable, from user simulation to statistical insights.

This research-focused, customer-facing role requires building automated insights, leading experimentation campaigns, and

Qualifications

  • Experience with LLM and AI agent evaluation methodologies.
  • Hands-on experience building or evaluating production-grade planner-executor systems.
  • Strong expertise in experimental design and statistical inference, including paired testing and confidence intervals.

Responsibilities

  • Design and lead an end-to-end evaluation framework for planner-executor agents.
  • Develop an LLM-powered user simulator and validate its behavior against production logs.
  • Build isolated, parallel experiment execution systems with configurable runs and adapters.
  • Instrument model calls and attribute token usage, latency, and costs to requests.
  • Establish grading methods and report KPIs like verified success rates and p95 latency.
  • Translate experimental results into projected financial impact and business value.
  • Lead staged experimentation campaigns and collaborate with engineers and business stakeholders.

Skills

LLM evaluation
Statistical inference
Experiment design
Python tooling
Communication with stakeholders
Mentoring engineers

Tools

LangFuse
LangSmith
Python

Job description

Jobgether, on behalf of a partner company, seeks a Principal Applied Scientist in the United States to lead the development of a rigorous evaluation framework for enterprise AI agents. You will collaborate with engineering teams to assess planner and executor models, and production-quality tools that make evaluation repeatable, from user simulation to statistical insights.

This research-focused, customer-facing role requires building automated insights, leading experimentation campaigns, and

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Enterprise AI Evaluation Lead & Methodology Scientist
Enterprise AI Evaluation Lead & Methodology Scientist

Jobgether • United States

Remote
USD 150,000 - 180,000
Principal Applied Scientist (Fully Remote)
Principal Applied Scientist (Fully Remote)

Jobgether • United States

Remote
USD 150,000 - 180,000
Senior AI Evaluation Scientist (Planner-Executor)
Senior AI Evaluation Scientist (Planner-Executor)

Robots & Pencils • United States

Remote
USD 200,000 - 300,000
Head of AI Quality & Insurance Ops (Remote)
Head of AI Quality & Insurance Ops (Remote)

Jobgether • United States

Remote
USD 135,000 - 145,000
Healthcare
401(k) matching
Paid parental leave
Principal Applied Scientist: Agentic AI for Enterprise
Principal Applied Scientist: Agentic AI for Enterprise

Amazon Web Services (AWS) • Seattle (WA)

On-site
USD 199,000 - 269,000
Senior Enterprise AI Strategist & Solution Architect
Senior Enterprise AI Strategist & Solution Architect

AI Technology Partners • United States

Remote
USD 200,000 - 260,000
Collaborative culture
Opportunity to shape enterprise AI
Meaningful client work
+1
Staff AI Product Engineer - Remote Enterprise Automation
Staff AI Product Engineer - Remote Enterprise Automation

Jobgether • United States

Remote
USD 200,000 - 300,000
Fully remote across continents
Stock options
Paid time off (18 days)
+1
Staff Engineer – Enterprise AI Evaluation & Automation
Staff Engineer – Enterprise AI Evaluation & Automation

Perplexity AI • New York (NY)

On-site
USD 150,000 - 230,000
Principal AI QA Engineer — Remote (GenAI Evaluation)
Principal AI QA Engineer — Remote (GenAI Evaluation)

BMC Software, Inc. • New York (NY)

Remote
USD 153,000 - 255,000
Remote Senior Enterprise AI Strategist & Architect
Remote Senior Enterprise AI Strategist & Architect

AI Technology Partners • Myrtle Point (OR)

Remote
USD 130,000 - 180,000
Competitive compensation package
Collaborative work environment
Opportunity for high-impact client work