Member of Technical Staff (Software Engineer, Enterprise Agent Evals & Efficiency)

Perplexity AI; New York (NY); On-site; USD 150,000 - 230,000; Full time

Job summary

Perplexity AI is seeking an experienced engineer to join the Enterprise Agent Evals & Efficiency team. You will own end-to-end evaluation environments, from building realistic scenarios to shipping improvements across the agent harness, tools, and skills.

Collaborate with AI Research, Connectors, and Platform teams to turn results into measurable gains in task success, latency, and cost. You will advance tool descriptions, contexts, and model routing strategies to ensure enterprise workflows are

Qualifications

• 5+ years of software or ML engineering with ownership of production systems
• Hands-on experience building/evaluating LLM-based applications or agents, including tool use and multi-step workflows
• Strong experimental and statistical judgment for meaningful task criteria and failure diagnosis
• Proficiency in a backend language such as Python, Go, or Rust

Responsibilities

• Build enterprise evaluation environments spanning connected applications, datasets, user roles, and permissions with multi-step workflows
• Own evaluation infrastructure, task suites, and automated graders calibrated against human judgment and outcomes
• Analyze agent trajectories and production failures to ship improvements across the harness, tools, and skills
• Develop reusable skills and improve tool descriptions, schemas, and result handling for better tool selection and error recovery
• Build model-selection and routing strategies balancing quality, latency, reliability, and cost

Skills

Python

Go

Rust

Software engineering

Machine learning engineering

Experimentation

Statistics

LLM/Agents

Tools

Python tooling

Production systems

Job description

• The Enterprise Agent Evals & Efficiency team makes Perplexity's agents better at completing real work across enterprise connectors, tools, and data. Our goal is to help agents choose the right tools, use the right context, and complete complex workflows reliably, with less time and compute. This team owns the improvement loop: define what successful work looks like, build evaluations that measure it, understand where agents fail, and ship changes that improve outcomes
• This is a hands-on engineering role with end-to-end ownership, from building realistic enterprise evaluation environments to shipping improvements across the agent harness, tools, and skills. You'll create complex scenarios spanning connected systems, data, and permissions, use them to identify where agents struggle, and improve the tools agents call, the skills they follow, the context they receive, and the models they use. Working with AI Research, Connectors, and Platform teams, you'll turn experimental results into measurable gains in task success, latency, and cost per successful task. The goal is not just to measure agents, but to make more enterprise work reliable and practical to automate
• Build realistic enterprise evaluation environments spanning connected applications, representative datasets, user roles, and permissions, with complex, multi-step workflows and reproducible system state
• Own evaluation infrastructure, task suites, and automated graders that measure how well agents complete enterprise work, calibrated against human judgment and grounded in end-to-end outcomes
• Analyze agent trajectories and production failures, identify root causes, and ship improvements across the agent harness, tools, and skills that measurably improve task success and reliability
• Develop reusable skills and improve tool descriptions, schemas, and result handling so agents can select the right tools, execute workflows correctly, and recover from errors with fewer unnecessary calls
• Build and evaluate model-selection and routing strategies that match each task to the right model and effort level, balancing quality, latency, reliability, and cost
• Improve agent efficiency through context management, skill loading, caching, and execution strategies that reduce wasted tokens and cost per successful task without compromising quality or safety
• Build regression gates and launch-readiness signals for model, connector, tool, skill, and harness changes, and validate that evaluation gains translate into better production outcomes

Hands-on experience building, evaluating, or improving LLM-based applications or agents, including tool use and multi-step workflowsStrong experimental and statistical judgment, including choosing representative tasks, defining meaningful success criteria, diagnosing failures, and distinguishing real improvements from noiseEvaluation infrastructure, benchmark design, and automated gradingStrong system design skills, including building reliable, scalable infrastructure for experimentation, evaluation, or data-intensive applicationsRealistic test environments or simulations spanning multiple systems, datasets, permissions, and stateful workflowsStrong proficiency in a backend language such as Python, Go, or Rust, and the ability to work effectively across a multi-language production stack5+ years of software engineering or machine learning engineering experience, with a track record of owning complex production systems from design through deployment and operationDeep expertise in at least one of the following:Model selection, routing, and inference efficiency, including context management and cachingAgent harnesses, tool interfaces, and reusable skills for complex workflowsA track record of leading ambiguous, cross-team initiatives from architecture through production adoption, setting technical direction, and raising engineering quality through mentorshipSound technical judgment in balancing task quality, latency, reliability, and cost, with a track record of translating evaluation findings into shipped improvements

We value demonstrated impact, deep ownership, and the initiative to turn ambiguity into results