Senior Software Engineer, AI-Driven Observability

HR Tech Job; Santa Clara (CA); Hybrid; USD 190,000 - 285,000; Full time

Job summary

Workday is hiring a Senior or Mid-Level Software Engineer for the Data Platform and Observability team in Santa Clara, CA. The role offers hybrid work with up to 50% remote and focuses on building and scaling an AI-powered observability platform.

You’ll design, implement, and operate monitoring, logging, alerting, and tracing services, leveraging Kubernetes, Docker, and cloud-native tools to support millions of messages daily while collaborating with product teams on SLOs/SLIs and production

Qualifications

• 8 years of software engineering/design experience.
• 7 years of coding in Python, Go, or Java.
• Experience with Linux.
• BS in Computer Science or related field, or equivalent experience.

Responsibilities

• Design, build, and improve critical observability services: Monitoring, Logging, Alerting, and Tracing.
• Own distributed tracing end to end — instrumenting services, propagating context across service boundaries, and using trace data to understand system behavior and diagnose issues in complex, multi-hop request paths.
• Build data capture and collection services using the latest technologies across multiple infrastructure types (Kubernetes, Docker, OpenStack, bare metal, etc.).
• Design and develop core software modules for real-time and batch data processing.
• Build metrics ingestion pipelines, alert definitions, and automation for dashboard lifecycle management.
• Instrument AI-powered and automated workflows for performance, cost, and quality, including multi-step processes and tool/service orchestration.
• Explore and apply AI-driven observability techniques — anomaly detection, intelligent alerting, and automated root-cause analysis — to reduce noise and speed up incident response.
• Partner with product and application teams to define SLOs/SLIs and production-readiness criteria for new services and automated workflows.
• Build infrastructure components and deploy them in production.
• Work across all aspects of observability with a keen eye for data quality, data integrity, and data availability.
• Evaluate and implement new open-source and cloud-native tools and technologies as needed.
• Participate in the on-call rotation supporting the observability platform.

Skills

Python

Go

Java

Linux

Education

BS in Computer Science or related field

MS Degree a plus

Tools

Docker

Kubernetes

Ansible

Chef

Terraform

Prometheus

Job description

Workday is hiring a Senior or Mid-Level Software Engineer for the Data Platform and Observability team in Santa Clara, CA. The role offers hybrid work with up to 50% remote and focuses on building and scaling an AI-powered observability platform.

You’ll design, implement, and operate monitoring, logging, alerting, and tracing services, leveraging Kubernetes, Docker, and cloud-native tools to support millions of messages daily while collaborating with product teams on SLOs/SLIs and production