Senior Distributed Systems Engineer - AI Observability

Workday; Santa Clara (CA); Hybrid; USD 190,000 - 285,000; Full time

Job summary

Workday is seeking a Senior/Mid-Level Software Engineer for the Data Platform and Observability team in Santa Clara. You will design and implement observability services, from monitoring to tracing, across diverse infrastructures.

The role offers a hybrid schedule with up to 50% remote work and collaboration with teams in Pleasanton, Dublin, and Boston. Candidates should have deep experience in distributed systems, Python/Go/Java, cloud platforms (AWS/GCP), and containerization, with a focus on

Qualifications

• 8 years of software engineering/design experience.
• 7 years of coding in Python, Go, or Java.
• Experience with Linux.
• BS in Computer Science or a related technical field, or equivalent experience.
• Design, maintain, and optimize Time-Series DBs, document stores, and log collection/management systems.
• Strong understanding of distributed tracing concepts and tooling.
• Public cloud experience (AWS/GCP), including native observability tools.
• Experience with containerization and infrastructure automation (Docker, Kubernetes, Ansible, Chef, Terraform).
• Experience with service mesh, Prometheus, and cloud-native technologies.
• Hands-on with LLM orchestration frameworks and agentic AI build patterns.
• Ability to spot AI production issues and build actionable dashboards/alerts.

Responsibilities

• Design, build, and improve observability services: Monitoring, Logging, Alerting, Tracing.
• Own distributed tracing end-to-end across services.
• Build data capture/collection services across infrastructures (K8s, Docker, OpenStack, etc.).
• Develop core software modules for real-time and batch data processing.
• Create metrics ingestion pipelines and automation for dashboards.
• Instrument AI-powered workflows for performance, cost, and quality.
• Apply AI-driven observability techniques to reduce noise and speed incidents.
• Collaborate with product teams to define SLOs/SLIs and production-readiness criteria.
• Contribute to production deployments and maintain data quality and availability.
• Evaluate new open-source/cloud-native tools as needed.

Skills

Software engineering

Python/Go/Java

Linux

Education

BS in Computer Science or related field

Tools

Kubernetes

Docker

Ansible

Chef

Terraform

Prometheus

AWS

GCP

LangChain

LlamaIndex

Semantic Kernel

Job description

Workday is seeking a Senior/Mid-Level Software Engineer for the Data Platform and Observability team in Santa Clara. You will design and implement observability services, from monitoring to tracing, across diverse infrastructures.

The role offers a hybrid schedule with up to 50% remote work and collaboration with teams in Pleasanton, Dublin, and Boston. Candidates should have deep experience in distributed systems, Python/Go/Java, cloud platforms (AWS/GCP), and containerization, with a focus on