Staff Software Engineer, Enterprise AI Evaluation

Perplexity; San Francisco (CA); On-site; USD 180,000 - 240,000; Full time

Job summary

Perplexity in San Francisco seeks an experienced Enterprise Agent Evals & Efficiency engineer to own end-to-end evaluation infrastructure and ship improvements across the agent harness, tools, and skills. You will create complex evaluation environments spanning connected systems and permissions, then measure outcomes and drive enhancements.

You will analyze agent trajectories, develop reusable skills, and build model-selection strategies that balance quality, latency, reliability, and cost

Qualifications

• 5+ years of software engineering or machine learning engineering experience.
• Strong system design skills for scalable experimentation and data-intensive apps.
• Proficiency in a backend language such as Python, Go, or Rust across multi-language stacks.
• Hands-on experience with LLM-based applications or agents and multi-step workflows.
• Expertise in evaluation infrastructure, agent harnesses, or realistic test environments.
• Experience leading cross-team initiatives from architecture to production adoption.

Responsibilities

• Build enterprise evaluation environments spanning connected apps, datasets, roles, and permissions.
• Own evaluation infrastructure, task suites, and automated graders aligned to human judgment and end-to-end outcomes.
• Analyze agent trajectories and failures, shipping improvements to harness, tools, and skills.
• Develop reusable skills and improve tool descriptions, schemas, and result handling for reliable workflows.
• Design model selection and routing strategies balancing quality, latency, reliability, and cost.
• Improve context management, caching, and execution strategies to reduce token usage and cost.

Skills

Software engineering

System design

Backend languages

LLM applications

Experimentation

Technical leadership

Job description

Perplexity in San Francisco seeks an experienced Enterprise Agent Evals & Efficiency engineer to own end-to-end evaluation infrastructure and ship improvements across the agent harness, tools, and skills. You will create complex evaluation environments spanning connected systems and permissions, then measure outcomes and drive enhancements.

You will analyze agent trajectories, develop reusable skills, and build model-selection strategies that balance quality, latency, reliability, and cost