Staff Software Engineer - Observability

EngineersOfAI; Amsterdam; On-site; EUR 120,000 - 190,000; Full time

Job summary

Adyen in Amsterdam is seeking a Staff Software Engineer to join the Platform Observability team. You will design and own telemetry data pipelines across logs, metrics and traces, ensuring reliability as billions of events flow daily.

You will lead strategies for Logging, Metrics, Tracing and Alerting, tackle scaling bottlenecks, and build tools to speed root-cause analysis while mentoring engineers and SREs to improve platform reliability and user experience.

Qualifications

• 5+ years of experience with highly distributed systems.
• 2+ years in building APIs and data pipelines for real-time data ingestion.
• Experience improving software reliability across availability, performance and incident management.
• Proficiency in Go and/or Java.
• Strong understanding of system design tradeoffs.
• Excellent stakeholder management and technical communication.
• Proficiency with Docker and Kubernetes; infrastructure as code (Terraform).
• Hands-on experience with core telemetry data stores and monitoring tooling.

Responsibilities

• Define and lead Logging, Metrics, Tracing & Alerting strategies.
• Resolve scaling bottlenecks in telemetry data pipelines.
• Create tooling to accelerate root-cause analysis and reduce MTTR.
• Mentor engineers and SREs on monitoring best practices.
• Identify bottlenecks and improve observability platform reliability.
• Lead features from ideation to adoption in the observability domain.
• Collaborate on planning to align engineering efforts and reduce complexity.

Skills

Distributed systems

APIs & data pipelines

Software reliability

Go/Java

System design

Stakeholder management

Docker & Kubernetes

Terraform

Observability tooling

OpenTelemetry

Grafana

Prometheus

Tools

Elasticsearch

Opensearch

ClickHouse

Job description

This is Adyen

Adyen provides payments, data, and financial products in a single solution for customers like Meta, Uber, H&M, and Microsoft - making us the financial technology platform of choice. At Adyen, everything we do is engineered for ambition.

For our teams, we create an environment with opportunities for our people to succeed, backed by the culture and support to ensure they are enabled to truly own their careers. We are motivated individuals who tackle unique technical challenges at scale and solve them as a team. Together, we deliver innovative and ethical solutions that help businesses achieve their ambitions faster.

Staff Software Engineer - Platform Observability

Adyen provides a global, unified platform with a wide variety of financial services. Observability plays a key role providing comprehensive data to a wide variety of use cases, ensuring the platform’s reliability and supporting our operations to run smoothly.

As part of the Observability team you will build and maintain products and services that enable engineers at Adyen to understand how their services are behaving in real-time, reliably diagnose issues and streamline data discovery through a common observability ecosystem. Between logs, metrics and traces, our platform collects and processes billions of events per day.

What you will do:

As a Staff Software Engineer working on the observability of the platform, you will play a key role in shaping how teams work with telemetry data, driving strategic decisions, identifying and advocating for best practices for building, running and maintaining observable components.

This includes:

• Define and lead Logging, Metrics, Tracing & Alerting strategies
• Solve scaling bottlenecks in critical services in our telemetry data pipelines
• Create advanced tooling to accelerate root-cause analysis, reduce Mean Time to Resolution (MTTR), and eliminate alert fatigue
• Guide and mentor software engineering and SRE teams on monitoring best practices and instrumenting code.
• Identify and tackle bottlenecks and single points of failures in the architecture aiming to improve the reliability of the observability platform and the overall user experience.
• Driving features and products idealization, implementation and adoption.
• Collaborate on planning and refinement proactively providing input whilst striving for engineering alignment, quality and reducing complexity and dependencies.
• Respond to alerts and duty responsibilities, like providing support to other engineers and troubleshooting production issues

What you bring:

• You have 5+ years of relevant work experience with highly distributed systems
• Expertise in designing and implementing APIs and data pipelines for high-throughput, real-time data ingestion
• Experience improving software reliability between different categories including availability, performance, latency, efficiency, capacity, SLOs and incident management.
• Proficiency developing and maintaining software and frameworks written in Go and/or Java
• Advanced understanding of system design and evaluating its tradeoffs
• Strong stakeholder management, technical communication, and mentorship capability.
• Highly motivated to learn and continuously develop yourself
• Experience with containerisation and orchestration technologies (Docker, Kubernetes) and infrastructure as code tools(e.g: Terraform)
• Observability Stack Expertise: You have hands-on experience operating core telemetry data stores at scale e.g. Elasticsearch/Opensearch/VictoriaLogs/Clickhouse for logging, Prometheus/ VictoriaMetrics for metrics and Grafana Tempo for distributed tracing, Grafana LGTM stack, OpenTelemetry, Alertmanager, Clickhouse

Nice to haves:

• Experience with highly available/fault tolerant, replicated data storage systems, large scale data processing systems is a strong plus
• Infrastructure and Platform Experience
• Contributions to open-source observabi