Member of Technical Staff (Software Engineer, Enterprise Agent Evals & Efficiency)

Perplexity AI

San Francisco (CA)

On-site

USD 180,000 - 260,000

Full time

14 days+

Get a reply from this employer

Your resume and cover letter, tailored to exactly what they’re hiring for.

Get past ATS filters

Job summary

Perplexity AI in San Francisco seeks an experienced Enterprise Agent Evals & Efficiency engineer to own evaluation infrastructure and end-to-end improvements for enterprise agents. You will design realistic workflows, measure outcomes, and ship tools, skills, and harness changes that boost task success and reduce costs.

You will collaborate with AI Research, Connectors, and Platform teams to drive measurable gains in latency, reliability, and automation of complex workflows.

Qualifications

  • 5+ years of software engineering or ML engineering experience, with ownership of production systems.
  • Strong system design skills for scalable experimentation infrastructure.
  • Proficient in Python, Go, or Rust and multi-language stacks.
  • Hands-on experience with LLM-based applications or agents and multi-step workflows.
  • Expertise in evaluation, agent harnesses, or realistic test environments.
  • Model selection, routing, and inference efficiency with context management.
  • Ability to translate evaluation findings into shipped improvements.

Responsibilities

  • Build realistic enterprise evaluation environments spanning connected applications, representative datasets, user roles, and permissions, with complex, multi-step workflows and reproducible system state.
  • Own evaluation infrastructure, task suites, and automated graders that measure how well agents complete enterprise work, calibrated against human judgment and grounded in end-to-end outcomes.
  • Analyze agent trajectories and production failures, identify root causes, and ship improvements across the agent harness, tools, and skills that measurably improve task success and reliability.
  • Develop reusable skills and improve tool descriptions, schemas, and result handling so agents can select the right tools, execute workflows correctly, and recover from errors with fewer unnecessary calls.
  • Build and evaluate model-selection and routing strategies that match each task to the right model and effort level, balancing quality, latency, reliability, and cost.
  • Improve agent efficiency through context management, skill loading, caching, and execution strategies that reduce wasted tokens and cost per successful task without compromising quality or safety.
  • Build regression gates and launch-readiness signals for model, connector, tool, skill, and harness changes, and validate that evaluation gains translate into better production outcomes.

Skills

Backend development
ML engineering
LLM applications
System design
Experimentation & measurement
Cross-team collaboration

Job description

ABOUT THE ROLE

The Enterprise Agent Evals & Efficiency team makes Perplexity's agents better at completing real work across enterprise connectors, tools, and data. Our goal is to help agents choose the right tools, use the right context, and complete complex workflows reliably, with less time and compute. This team owns the improvement loop: define what successful work looks like, build evaluations that measure it, understand where agents fail, and ship changes that improve outcomes. This is a hands-on engineering role with end-to-end ownership, from building realistic enterprise evaluation environments to shipping improvements across the agent harness, tools, and skills. You'll create complex scenarios spanning connected systems, data, and permissions, use them to identify where agents struggle, and improve the tools agents call, the skills they follow, the context they receive, and the models they use. Working with AI Research, Connectors, and Platform teams, you'll turn experimental results into measurable gains in task success, latency, and cost per successful task. The goal is not just to measure agents, but to make more enterprise work reliable and practical to automate.

KEY RESPONSIBILITIES
  • Build realistic enterprise evaluation environments spanning connected applications, representative datasets, user roles, and permissions, with complex, multi-step workflows and reproducible system state.
  • Own evaluation infrastructure, task suites, and automated graders that measure how well agents complete enterprise work, calibrated against human judgment and grounded in end-to-end outcomes.
  • Analyze agent trajectories and production failures, identify root causes, and ship improvements across the agent harness, tools, and skills that measurably improve task success and reliability.
  • Develop reusable skills and improve tool descriptions, schemas, and result handling so agents can select the right tools, execute workflows correctly, and recover from errors with fewer unnecessary calls.
  • Build and evaluate model-selection and routing strategies that match each task to the right model and effort level, balancing quality, latency, reliability, and cost.
  • Improve agent efficiency through context management, skill loading, caching, and execution strategies that reduce wasted tokens and cost per successful task without compromising quality or safety.
  • Build regression gates and launch-readiness signals for model, connector, tool, skill, and harness changes, and validate that evaluation gains translate into better production outcomes.
QUALIFICATIONS
  • 5+ years of software engineering or machine learning engineering experience, with a track record of owning complex production systems from design through deployment and operation.
  • Strong system design skills, including building reliable, scalable infrastructure for experimentation, evaluation, or data-intensive applications.
  • Strong proficiency in a backend language such as Python, Go, or Rust, and the ability to work effectively across a multi-language production stack.
  • Hands-on experience building, evaluating, or improving LLM-based applications or agents, including tool use and multi-step workflows.
  • Deep expertise in at least one of the following:
    • Evaluation infrastructure, benchmark design, and automated grading.
    • Agent harnesses, tool interfaces, and reusable skills for complex workflows.
    • Realistic test environments or simulations spanning multiple systems, datasets, permissions, and stateful workflows.
  • Model selection, routing, and inference efficiency, including context management and caching.
  • Strong experimental and statistical judgment, including choosing representative tasks, defining meaningful success criteria, diagnosing failures, and distinguishing real improvements from noise.
  • Sound technical judgment in balancing task quality, latency, reliability, and cost, with a track record of translating evaluation findings into shipped improvements.
  • A track record of leading ambiguous, cross-team initiatives from architecture through production adoption, setting technical direction, and raising engineering quality through mentorship.

If you’re excited about this role, we encourage you to apply even if you don’t meet every qualification. We value demonstrated impact, deep ownership, and the initiative to turn ambiguity into results.

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Member of Technical Staff (Software Engineer, Enterprise Agent Evals & Efficiency)
Member of Technical Staff (Software Engineer, Enterprise Agent Evals & Efficiency)

FDE Times • San Francisco (CA)

On-site
USD 180,000 - 240,000
Member of Technical Staff (Software Engineer, Enterprise Agent Evals & Efficiency)
Member of Technical Staff (Software Engineer, Enterprise Agent Evals & Efficiency)

Perplexity AI • New York (NY)

On-site
USD 150,000 - 230,000
Member of Technical Staff (Software Engineer, Enterprise Agent Evals & Efficiency)
Member of Technical Staff (Software Engineer, Enterprise Agent Evals & Efficiency)

Perplexity • San Francisco (CA)

On-site
USD 180,000 - 240,000
Member of Technical Staff (Applied AI Engineer, Agent Capabilities)
Member of Technical Staff (Applied AI Engineer, Agent Capabilities)

FDE Times • San Francisco (CA)

On-site
USD 180,000 - 280,000
Staff Software Engineer, Agent Eval Platform
Staff Software Engineer, Agent Eval Platform

ServiceNow • Santa Clara (CA)

On-site
USD 180,000 - 320,000
Staff Software Engineer, Enterprise AI Evaluation
Staff Software Engineer, Enterprise AI Evaluation

Perplexity • San Francisco (CA)

On-site
USD 180,000 - 240,000
Member of Technical Staff (Software Engineer, Enterprise Knowledge Platform)
Member of Technical Staff (Software Engineer, Enterprise Knowledge Platform)

Perplexity • Palo Alto (CA)

On-site
USD 180,000 - 260,000
Staff Engineer, Enterprise AI Evaluation & Automation
Staff Engineer, Enterprise AI Evaluation & Automation

Fdetimes • San Francisco (CA)

On-site
USD 180,000 - 240,000
Software Engineer (Agent Evaluation Platform)
Software Engineer (Agent Evaluation Platform)

ServiceNow • Mountain View (CA)

On-site
USD 130,000 - 210,000
Generous family leave
Matched donations
Annual learning stipends
+3
Staff Engineer – Enterprise AI Evaluation & Automation
Staff Engineer – Enterprise AI Evaluation & Automation

Perplexity AI • New York (NY)

On-site
USD 150,000 - 230,000