Remote Principal Software Engineer, AI Observability

Palo Alto Networks, Inc.; United States; Remote; USD 125,000 - 188,000; Full time

Job summary

Chronosphere, a Palo Alto Networks company, is building AI Observability capabilities on its modern observability platform. We focus on testing, measuring, and benchmarking AI agents and RAG applications before deployment and in production.

You’ll work with product managers and customers to translate needs into scalable backend systems and workflows. We are seeking engineers with strong product sense who ask why as often as how.

Qualifications

• 8+ years of software engineering experience.
• Strong experience with backend languages (Go/Java/Python/Rust).
• Experience designing and operating APIs (REST, gRPC, GraphQL).
• Hands-on with cloud platforms (AWS, GCP, Azure).
• Experience building AI observability/evaluation features is a plus.
• Experience delivering production-grade, scalable systems.
• Ability to work with cross-functional teams and stakeholders.
• Self-driven with good judgement on when to push forward or ask for help.

Responsibilities

• Build backend capabilities to observe and evaluate AI models and RAG apps.
• Design systems for testing, measurement, and benchmarking of AI components.
• Develop scalable APIs and data processing to contextualize AI telemetry.
• Help shape roadmap and architecture for AI Observability on Chronosphere platform.
• Collaborate with PMs, engineers, and customers to deliver integrated features.
• Improve system observability using our own product.
• Enhance performance, reliability, and cost efficiency of distributed AI tooling.
• Mentor engineers and uphold engineering standards.

Skills

Backend languages Go/Java/Python/Rust

8+ years software engineering

Cloud infrastructure (AWS/GCP/Azure)

API design (REST/gRPC/GraphQL)

LLM-powered features experience

Production-ready systems

Cross-functional collaboration

Autonomous working style

Job description

Chronosphere, a Palo Alto Networks company, is building AI Observability capabilities on its modern observability platform. We focus on testing, measuring, and benchmarking AI agents and RAG applications before deployment and in production.

You’ll work with product managers and customers to translate needs into scalable backend systems and workflows. We are seeking engineers with strong product sense who ask why as often as how.