Principal Software Development Engineer - Observability

Expedia Group; San Jose (CA); On-site; USD 249,000 - 348,500; Full time

Benefits offered by this job

Medical, dental, and vision coverage

Travel discounts

Wellness and travel reimbursement

Job summary

Expedia Group is seeking a Principal Software Engineer for Observability in San Jose. You will lead architecture and delivery of a centralized observability platform, handling logs, metrics and traces for thousands of services.

You will champion OpenTelemetry adoption, establish governance and cost controls, and unify tooling across teams while mentoring engineers and driving production reliability in a complex, cloud-native environment.

Qualifications

• Bachelor’s or Master’s degree in Computer Science or a related technical field, or equivalent practical experience.
• 10+ years of experience in software engineering, with a focus on building and operating large-scale distributed systems, infrastructure automation, or configuration management.
• Deep expertise in observability principles and the three pillars: logs, metrics, and traces.
• Strong hands-on proficiency with observability technologies such as Prometheus, Grafana, Datadog, Splunk, and OpenTelemetry.
• Proficiency in one or more of Go, Java, Python.
• Solid understanding of cloud-native architectures (Kubernetes, Docker, microservices) and major cloud platforms (AWS preferred).

Responsibilities

• Architect and build core telemetry pipelines: scalable pipelines for logs, metrics, traces and cost-effective scaling for 10x data growth.
• Drive OpenTelemetry adoption: strategy, rollout, and automated configurations for data collection across services.
• Implement platform governance and optimization: data governance, cost allocation, and SLOs for the platform; tooling for observability costs.
• Elevate observability practice: unify tooling (Grafana, Datadog, Splunk), documentation, and service lifecycle in the internal portal.
• Automate infrastructure lifecycle: IaC with Terraform and/or Crossplane; automate cluster provisioning and upgrades.
• Provide technical leadership and mentorship: architecture reviews and RFCs; mentor senior engineers.
• Production debugging: escalate complex incidents across the observability platform.
• Collaborate and innovate: work with Go, Java, Python, AWS, Kubernetes, OpenTelemetry, Prometheus, Grafana, Datadog, Splunk, Clickhouse.

Skills

Observability

OpenTelemetry

Prometheus

Grafana

Datadog

Splunk

Go

Java

Python

Kubernetes

Docker

AWS

Education

Bachelor’s or Master’s degree in Computer Science or related field

Tools

OpenTelemetry

Prometheus

Grafana

Datadog

Splunk

Kubernetes

Docker

Clickhouse

Job description

At Expedia Group, we help travelers explore the world, one journey at a time. As a global travel company powered by passionate people, trusted partnerships, and leading technology, we connect travelers, partners, and advertisers through our consumer brands, B2B network, and travel advertising business.

Here, you'll do meaningful work that helps millions of people discover, book, and experience travel with more ease, confidence, and joy. Our five Behaviors-Traveler First, Think Big, Operate with Excellence, Ownership Mindset, and Succeed Together-help foster a supportive environment where people can grow their careers and have the flexibility, benefits, and support to do their best work. Join us and build for travelers everywhere.

Principal Software Engineer, Observability

Introduction to the Team:

Our Technology Team partners with teams across Expedia Group to create innovative products, services, and tools to deliver high-quality experiences for travelers, partners, and our employees.A singular technology platform powered by data and machine learning provides secure, differentiated, and personalized experiences that drive loyalty and traveler satisfaction.

As a Principal Engineer, you will be part of an agile development team with deep expertise in cloud, distributed systems, and observability. You will play a pivotal role in crafting the strategic technical goals for our group. The main effort will involve leading the architecture, design, and implementation of a centralized, scalable, and cost-effective observability platform used by all engineering teams across Expedia.

You will provide technical leadership for a dynamic engineering organization and work alongside talented product managers and other technical leaders to deliver best-in-class capabilities to our developer community.

In this role, you will:

• Architect and Build Core Telemetry Pipelines: Lead the design and implementation of highly scalable and resilient telemetry pipelines for logs, metrics, and traces. Evolve our platform to handle a 10x increase in data volume while maintaining performance and cost-effectiveness
• Drive OpenTelemetry Adoption: Spearhead the strategy, rollout, and support for the OpenTelemetry collector across thousands of services. Develop best practices and automated configurations to ensure seamless and consistent data collection
• Implement Platform Governance and Optimization: Design and build capabilities for data governance, cost allocation, and resource management within the observability platform. Define and implement SLOs for the platform itself and create tools to help teams manage their observability costs
• Elevate the Practice of Observability: Act as a thought leader, driving the adoption of observability best practices across the engineering organization. Improve the developer experience by unifying tooling (e.g., Grafana, Datadog, Splunk), documentation, and service lifecycle management within our internal developer portal
• Automate Infrastructure Lifecycle: Author and maintain production-grade Infrastructure as Code (IaC) using tools like Terraform and/or Crossplane. Eliminate manual toil by automating cluster provisioning, dependency upgrades, and incident remediation workflows
• Technical Leadership and Mentorship: Act as a force multiplier. Mentor senior engineers on the team, lead architecture review sessions, and author RFCs to build consensus on significant technical decisions. Your influence will extend beyond the team to application developers and SREs
• Production Debugging: Serve as the final escalation point for complex, cross-cutting production incidents related to the observability platform, from telemetry agent bugs to data correlation failures in our distributed systems
• Collaborate and Innovate: Explore and utilize a wide variety of technologies and tools, such as (but not limited to) Go, Java, Python, AWS, Kubernetes, OpenTelemetry, Prometheus, Grafana, Datadog, and Splunk, Clickhouse

Minimum Qualifications:

• Bachelor’s or Master’s degree in Computer Science or a related technical field, or equivalent practical experience
• 10+ years of experience in software engineering, with a focus on building and operating large-scale distributed systems, infrastructure automation, or configuration management
• Deep expertise in observability principles and the "three pillars": logs, metrics, and traces
• Strong hands-on proficiency with observability technologies such as Prometheus, Grafana, Datadog, Splunk, and OpenTelemetry
• Proficient in one or more of: Go, Java, Python
• Solid understanding of cloud-native architectures (Kubernetes, Docker, microservices) and major cloud platforms (AWS preferred)

Preferred Qualifications:

• Experience designing, building, and operating highly available, scalable, and resilient platforms
• Excellent hands-on coder who understands and appreciates bigger-picture architectural and business concerns
• Clear communicator with the ability to concisely explain complex technical details to a wide variety of audiences in both verbal and written form
• A creative problem solver who uses data and insights to support recommendations and influence decisions
• Experience mentoring other senior engineers and establishing standards for operational excellence and code quality at a multi-project level

The total cash range for this position in San Jose is $249,000.00 to $348,500.00. Employees in this role have the potential to increase their pay up to $398,500.00, which is the top of the range, based on ongoing, demonstrated, and sustained performance in the role. The total cash range for this position in Seattle is $231,000.00 to $323,500.00. Employees in this role have the potential to increase their pay up to $369,500.00, which is the top of the range, based on ongoing, demonstrated, and sustained performance in the role.

Starting pay for this role will vary based on multiple factors, including location, available budget, and an individual’s knowledge, skills, and experience. Pay ranges may be modified in the future.

Benefits And Perks

• Expedia Group offers benefits and perks designed to support employees and their families, including medical, dental, and vision coverage, paid time off, an Employee Assistance Program, wellness and travel reimbursement, travel discounts, and International Airlines Travel Agent Network (IATAN) membership. Learn more about life at Expedia Group at https://careers.expediagroup.com/life.

Accommodation requests

Expedia Group is committed to providing an inclusive and accessible recruiting experience. If you need an accommodation or adjustment due to a disability during the application or recruiting process, please submit a request at https://expedia.service-now.com/askeg?id=job_accommodation.

About Expedia Group

Expedia Group includes three flagship consumer brands - Expedia, Hotels.com, and Vrbo - along with a leading B2B travel business and travel advertising offerings. Across our brands and business, we help travelers explore the world with confidence and ease.

Equal Opportunity

Expedia is committed to creating an inclusive work environment with a diverse workforce.

All qualified applicants will receive consideration for employment without regard to race, color, religion, gender, gender identity or expression, sexual orientation, national origin, genetics, disability, age, veteran status, or any other characteristic protected by law.

This employer participates in E-Verify.

The employer will provide the Social Security Administration (SSA) and, if necessary, the Department of Homeland Security (DHS) with information from each new employee's I-9 to confirm work authorization.