Principal Observability Engineer: Tracing & AI Platform

Workday, Inc.; Pleasanton (CA); Hybrid; USD 223,000 - 334,000; Full time

Benefits offered by this job

Flexible work options

Job summary

Workday, Inc. is seeking a senior engineering leader to own the technical vision for the distributed tracing portion of our Observability Platform.

You’ll architect and build a multi-petabyte, low-latency tracing system on ClickHouse/Tempo with a big-data pipeline (Kafka, Spark/Flink, Iceberg) on AWS, and guide multiple teams toward scalable, reliable operations. You will mentor senior engineers, drive architecture decisions, and push for AI-driven incident triage and anomaly detection across

Qualifications

• 14+ years experience in software development engineering.
• 6+ years designing, building, and operating complex distributed systems.
• 8+ years experience with two or more of Java, Python, Go, including production-grade distributed systems.
• Bachelor’s degree in Computer Science, Engineering, or related field; Master’s preferred or equivalent practical experience.

Responsibilities

• Own the technical vision and architecture for distributed tracing within Workday's Observability Platform.
• Design and build multi-petabyte, low-latency tracing infrastructure end-to-end.
• Define direction across teams; mentor senior and staff engineers; act as primary architect for tracing subsystem.
• Lead ingestion/storage design, query performance, and platform reliability for tracing data.

Skills

Java

Python

Go

Distributed systems

API development

Big data processing

Education

Bachelor's degree in CS/Engineering

Master's degree preferred

Tools

Kafka

Spark/Flink

Iceberg

ClickHouse

Tempo

S3

Grafana

Job description

Workday, Inc. is seeking a senior engineering leader to own the technical vision for the distributed tracing portion of our Observability Platform.

You’ll architect and build a multi-petabyte, low-latency tracing system on ClickHouse/Tempo with a big-data pipeline (Kafka, Spark/Flink, Iceberg) on AWS, and guide multiple teams toward scalable, reliable operations. You will mentor senior engineers, drive architecture decisions, and push for AI-driven incident triage and anomaly detection across