Principal Engineer, Software - Backend, AI Observability/Evaluations (Chronosph

Palo Alto Networks, Inc.; United States; Remote; USD 125,000 - 188,000; Full time

Job summary

Chronosphere, a Palo Alto Networks company, is building AI Observability capabilities on its modern observability platform. We focus on testing, measuring, and benchmarking AI agents and RAG applications before deployment and in production.

You’ll work with product managers and customers to translate needs into scalable backend systems and workflows. We are seeking engineers with strong product sense who ask why as often as how.

Qualifications

• 8+ years of software engineering experience.
• Strong experience with backend languages (Go/Java/Python/Rust).
• Experience designing and operating APIs (REST, gRPC, GraphQL).
• Hands-on with cloud platforms (AWS, GCP, Azure).
• Experience building AI observability/evaluation features is a plus.
• Experience delivering production-grade, scalable systems.
• Ability to work with cross-functional teams and stakeholders.
• Self-driven with good judgement on when to push forward or ask for help.

Responsibilities

• Build backend capabilities to observe and evaluate AI models and RAG apps.
• Design systems for testing, measurement, and benchmarking of AI components.
• Develop scalable APIs and data processing to contextualize AI telemetry.
• Help shape roadmap and architecture for AI Observability on Chronosphere platform.
• Collaborate with PMs, engineers, and customers to deliver integrated features.
• Improve system observability using our own product.
• Enhance performance, reliability, and cost efficiency of distributed AI tooling.
• Mentor engineers and uphold engineering standards.

Skills

Backend languages Go/Java/Python/Rust

8+ years software engineering

Cloud infrastructure (AWS/GCP/Azure)

API design (REST/gRPC/GraphQL)

LLM-powered features experience

Production-ready systems

Cross-functional collaboration

Autonomous working style

Job description

Location: Australia

Our Mission

At Palo Alto Networks we're united by a shared mission—to protect our digital way of life. We thrive at the intersection of innovation and impact, solving real-world problems with cutting-edge technology and bold thinking. Here, everyone has a voice, and every idea counts. If you're ready to do the most meaningful work of your career alongside people who are just as passionate as you are, you're in the right place.

Who We Are

In order to be the cybersecurity partner of choice, we must trailblaze the path and shape the future of our industry. This is something our employees work at each day and is defined by our values: Disruption, Collaboration, Execution, Integrity, and Inclusion. We weave AI into the fabric of everything we do and use it to augment the impact every individual can have. If you are passionate about solving real-world problems and ideating beside the best and the brightest, we invite you to join us!

This role is remote, but distance is no barrier to impact. Our hybrid teams collaborate across geographies to solve big problems, stay close to our customers, and grow together. You will be part of a culture that values trust, accountability, and shared success where your work truly matters.

Job Summary

About us

Chronosphere, a Palo Alto Networks company, is the observability platform built for control in the modern, containerized world. Chronosphere empowers customers to focus on the data and insights that matter by reducing complexity, optimizing costs, and remediating issues faster. Chronosphere reduces data volumes and associated costs by 84% on average while saving developers thousands of hours. Recognized as a leader by major analyst firms, Chronosphere is trusted by the world's most innovative brands, including DoorDash, affirm, and Zillow.

About our roles

We are building AI Observability and Evaluation capabilities on Chronosphere's observability platform to help developers understand and improve the accuracy, relevance, performance, and safety of AI models, agents, and retrieval-augmented generation (RAG) applications. Our team enables customers to systematically test, measure, and benchmark AI agents and applications before deployment and monitor how they behave in production. We turn complex telemetry and evaluation data into insights that help developers build and operate reliable AI agents and applications at scale.

We are looking for experienced engineers with a strong product sense who asks "why" as often as "how". You'll partner with product managers, engineers, and customers to translate emerging AI observability and evaluation needs into scalable backend systems and cohesive developer workflows. You'll help shape the technical roadmap and architecture, taking features from early exploration through production delivery.

Qualifications

You will

• Build backend capabilities that help developers observe and evaluate AI models, agents, and RAG applications before and during production
• Design systems for systematic testing, measurement, and benchmarking of AI agents and applications accuracy, relevance, performance, and safety
• Build scalable APIs and data processing systems that contextualize AI telemetry and evaluation results, helping customers understand application behavior and identify opportunities for improvement
• Help shape the technical roadmap and architecture for AI Observability and Evaluation, building on Chronosphere's rapidly scaling platform
• Collaborate with product managers, deeply technical teammates, and customers to scope, validate, and deliver integrated features and developer workflows
• Improve our own system observability (we use our own product heavily)
• Improve the performance, reliability, and cost efficiency of the distributed systems powering AI Observability and Evaluation
• Mentor engineers and help maintain our engineering standards

You have

• 8+ years of experience in software engineering
• Strong experience in one or more backend languages (e.g., Go, Java, Python, Rust)
• Some experience developing LLM powered product features is desirable
• Experience using AI coding assistants (e.g., Cursor, Codex, Claude) as part of day-to-day software development
• Experience building and operating highly available systems in production
• Experience working with cloud infrastructure (AWS, GCP, Azure)
• Experience designing APIs (REST, gRPC, GraphQL) for scale
• Experience working effectively with cross functional partners
• Experience owning and executing projects from inception to delivery
• Experience working with a high degree of autonomy with good judgement on when to push forward and when to ask for help
• An effective collaborative working style and willingness to build and