Senior Distributed Systems Engineer - AI-Driven Observability

Workday, Inc.; Santa Clara (CA); Hybrid; USD 190,000 - 285,000; Full time

Job summary

Workday, Inc. is seeking a Senior or Mid-Level Software Engineer for the Data Platform && Observability team.

The role focuses on building high-scale observability services, including monitoring, logging, alerting, and tracing, with end-to-end tracing ownership across multi-hop paths. You will work with Kubernetes, Docker, OpenStack, and modern cloud-native tools, applying AI-driven approaches to improve anomaly detection, alerting, and root-cause analysis in production environments.

Qualifications

• 8 years of software engineering/design experience.
• 7 years of coding in Python, Go, or Java.
• Experience with Linux.
• BS in Computer Science or a related technical field, or equivalent experience.
• Public cloud experience (AWS/GCP), including native observability tooling.
• Experience with containerization and infrastructure automation (Docker, Kubernetes, Ansible, Chef, Terraform).
• Experience with service mesh, Prometheus, and cloud-native technologies.
• Hands-on experience with LLM orchestration frameworks (e.g., LangChain, LlamaIndex, Semantic Kernel).
• A knack for spotting AI production issues and building measurable signals.

Responsibilities

• Design, build, and improve critical observability services: Monitoring, Logging, Alerting, and Tracing.
• Own distributed tracing end to end — instrumenting services and diagnosing issues across multi-hop paths.
• Build data capture and collection services across infrastructure types (Kubernetes, Docker, OpenStack, bare metal).
• Design and develop core software modules for real-time and batch data processing.
• Build metrics ingestion pipelines, alert definitions, and dashboard lifecycle automation.
• Instrument AI-powered workflows for performance, cost, and quality and integrate tool orchestration.
• Explore AI-driven observability techniques to reduce noise and speed up incident response.
• Collaborate with product and application teams to define SLOs/SLIs and production-readiness criteria.
• Build infrastructure components and deploy them in production.
• Participate in on-call rotations for observability platform.

Skills

Python

Go

Java

Linux

AWS

GCP

Docker

Kubernetes

Ansible

Chef

Terraform

Prometheus

LangChain

LlamaIndex

Semantic Kernel

Time-Series DBs

Distributed tracing

Education

BS in Computer Science or related field

MS Degree a plus

Tools

AWS CloudWatch

GCP Stackdriver

Job description

Workday, Inc. is seeking a Senior or Mid-Level Software Engineer for the Data Platform && Observability team.

The role focuses on building high-scale observability services, including monitoring, logging, alerting, and tracing, with end-to-end tracing ownership across multi-hop paths. You will work with Kubernetes, Docker, OpenStack, and modern cloud-native tools, applying AI-driven approaches to improve anomaly detection, alerting, and root-cause analysis in production environments.