Staff Engineer, AI Evaluation & Observability

Pantera Capital; Palo Alto (CA); On-site; USD 150,000 - 210,000; Full time

Job summary

Perplexity is seeking an engineer to advance its evaluation infrastructure and production-grade systems supporting Answer Quality. You will collaborate with data scientists, engineers, and product teams to identify quality problems, measure their impact, and drive improvements across prompts, tools, search systems, datasets, and models.

You will build scalable pipelines for processing and analyzing large-scale interaction traces, ensuring reliability, observability, and actionable insights for

Qualifications

• 4+ years of software, data, or machine learning engineering experience shipping and operating production systems.
• Strong proficiency in Python and SQL, with fundamentals in system design, data modeling, and distributed systems.
• Experience building big-data systems, including distributed compute, large-scale storage, and high-volume pipelines.

Responsibilities

• Build shared evaluation infrastructure that helps teams run reliable evals, analyze results, and make product and model decisions.
• Develop the platform for replaying and analyzing agent traces to reproduce production behavior and diagnose failures.
• Build and operate scalable systems for processing, storing, and monitoring interaction, trace, and evaluation data.
• Partner with data scientists, engineers, and product teams to turn answer-quality problems into evaluations, analyses, and product improvements.
• Operate in a small, high-impact team where your work directly shapes how Perplexity measures and improves Answer Quality.

Skills

Python

SQL

Distributed systems

Data modeling

Big-data

Machine learning engineering

Production systems

Tools

Databricks

Snowflake

ClickHouse

Job description

Perplexity is seeking an engineer to advance its evaluation infrastructure and production-grade systems supporting Answer Quality. You will collaborate with data scientists, engineers, and product teams to identify quality problems, measure their impact, and drive improvements across prompts, tools, search systems, datasets, and models.

You will build scalable pipelines for processing and analyzing large-scale interaction traces, ensuring reliability, observability, and actionable insights for