Sr Principal / Architect Engineer, Software - Backend, AI Observability/Evaluations (Chronosphere)

Palo Alto Networks; United States; Remote; USD 210,000 - 320,000; Full time

Job summary

Palo Alto Networks is seeking an experienced backend engineer to help build AI Observability and Evaluation capabilities on Chronosphere’s platform. You will design scalable backend systems and APIs to observe AI models, agents, and RAG applications, and collaborate with product managers and customers to translate needs into reliable software deployments.

The role requires strong autonomy in a remote-first environment, mentoring peers, and improving observability and cost efficiency across

Qualifications

• 12-15+ years of software engineering experience.
• Strong experience with one or more backend languages (Go, Java, Python, Rust).
• Experience designing and consuming scalable APIs (REST, gRPC, GraphQL).
• Experience building and operating highly available production systems.
• Familiarity with cloud platforms (AWS, GCP, Azure).
• Experience using AI coding assistants (Cursor, Codex, Claude).
• Ability to work autonomously in a remote-first culture.

Responsibilities

• Build backend capabilities to observe and evaluate AI models, agents, and RAG apps.
• Design systems for testing, measurement, and benchmarking of AI telemetry and results.
• Build scalable APIs and data processing systems for AI observability.
• Help shape the roadmap and architecture for AI Observability and Evaluation.
• Collaborate with product managers and customers to deliver features.
• Improve system observability and cost efficiency.
• Mentor engineers and maintain engineering standards.

Skills

Backend languages

Distributed systems

APIs design

Cloud infrastructure

Remote collaboration

Tools

Cursor

Codex

Claude

Job description

Our Mission

At Palo Alto Networks®, we’re united by a shared mission—to protect our digital way of life. We thrive at the intersection of innovation and impact, solving real-world problems with cutting-edge technology and bold thinking. Here, everyone has a voice, and every idea counts. If you’re ready to do the most meaningful work of your career alongside people who are just as passionate as you are, you’re in the right place.

Who We Are

In order to be the cybersecurity partner of choice, we must trailblaze the path and shape the future of our industry. This is something our employees work at each day and is defined by our values: Disruption, Collaboration, Execution, Integrity, and Inclusion. We weave AI into the fabric of everything we do and use it to augment the impact every individual can have. If you are passionate about solving real-world problems and ideating beside the best and the brightest, we invite you to join us!

This role is remote, but distance is no barrier to impact. Our hybrid teams collaborate across geographies to solve big problems, stay close to our customers, and grow together. You will be part of a culture that values trust, accountability, and shared success where your work truly matters.

Job Summary

About us

Chronosphere, a Palo Alto Networks company, is the observability platform built for control in the modern, containerized world. Chronosphere empowers customers to focus on the data and insights that matter by reducing complexity, optimizing costs, and remediating issues faster. Chronosphere reduces data volumes and associated costs by 84% on average while saving developers thousands of hours. Recognized as a leader by major analyst firms, Chronosphere is trusted by the world’s most innovative brands, including DoorDash, ... and Zillow.

About our roles

We are building AI Observability and Evaluation capabilities on Chronosphere’s observability platform to help developers understand and improve the accuracy, relevance, performance, and safety of AI models, agents, and retrieval-augmented generation (RAG) applications. Our team enables customers to systematically test, measure, and benchmark AI agents and applications before deployment and monitor how they behave in production. We turn complex telemetry and evaluation data into insights that help developers build and operate reliable AI agents and applications at scale.

We are looking for experienced engineers with a strong product sense who asks “why” as often as “how”. You’ll partner with product managers, engineers, and customers to translate emerging AI observability and evaluation needs into scalable backend systems and cohesive developer workflows. You’ll help shape the technical roadmap and architecture, taking features from early exploration through production delivery.

Qualifications

You will

• Build backend capabilities that help developers observe and evaluate AI models, agents, and RAG applications before and during production
• Design systems for systematic testing, measurement, and benchmarking of AI agents and applications accuracy, relevance, performance, and safety
• Build scalable APIs and data processing systems that contextualize AI telemetry and evaluation results, helping customers understand application behavior and identify opportunities for improvement
• Help shape the technical roadmap and architecture for AI Observability and Evaluation, building on Chronosphere’s rapidly scaling platform
• Collaborate with product managers, deeply technical teammates, and customers to scope, validate, and deliver integrated features and developer workflows
• Improve our own system observability (we use our own product heavily)
• Improve the performance, reliability, and cost efficiency of the distributed systems powering AI Observability and Evaluation
• Mentor engineers and help maintain our engineering standards

You have

• 12-15+ years of experience in software engineering
• Strong experience in one or more backend languages (e.g., Go, Java, Python, Rust)
• Some experience developing LLM powered product features is desirable
• Experience using AI coding assistants (e.g., Cursor, Codex, Claude) as part of day-to-day software development
• Experience building and operating highly available systems in production
• Experience working with cloud infrastructure (AWS, GCP, Azure)
• Experience designing APIs (REST, gRPC, GraphQL) for scale
• Experience working effectively with cross functional partners
• Experience owning and executing projects from inception to delivery
• Experience working with a high degree of autonomy with good judgement on when to push forward and when to ask for help
• An effective collaborative working style and willingness to build and maintain a positive and supportive environment
• An ability to operate effectively in a remote first environment

Our Commitment

We’re trailblazers that dream big, take risks, and challenge cybersecurity’s status quo. It’s simple: we can’t accomplish our mission without diverse teams innovating, together.

We are committed to providing reasonable accommodations for all qualified individuals with a disability. If you require assistance or accommodation due to a disability or special need, please contact us at [e-mail].

Palo Alto Networks is an equal opportunity employer. We celebrate diversity in our workplace, and all qualified applicants will receive consideration for employment without regard to age, ancestry, color, family or medical care leave, gender identity or expression, genetic information, marital status, medical condition, national origin, physical or mental disability, political affiliation, protected veteran status, race, religion, sex (including pregnancy), sexual orientation, or other legally protected characteristics.

All your information will be kept confidential according to EEO guidelines.

Is role eligible for Immigration Sponsorship?: Yes