Staff Software Engineer - Observability

United States Digital Space LLC; Amsterdam; On-site; EUR 110,000 - 140,000; Full time

Job summary

United States Digital Space LLC in Amsterdam is seeking a Staff Software Engineer for Platform Observability to help design and scale our telemetry data pipelines. You will work on logging, metrics, tracing, and alerting to ensure reliability across billions of events daily.

You will mentor teams, define strategies for logging, metrics, and tracing, and collaborate with SRE to reduce MTTR, improve performance, and maintain a highly available observability platform.

Qualifications

• 5+ years of experience with highly distributed systems.
• Experience designing APIs and data pipelines for real-time data ingestion.
• Proven ability to improve software reliability including latency, availability, and incident management.
• Proficiency in Go and/or Java.
• Strong system design skills and ability to evaluate tradeoffs.
• Excellent stakeholder management and technical communication.

Responsibilities

• Define and lead Logging, Metrics, Tracing & Alerting strategies.
• Solve scaling bottlenecks in telemetry data pipelines.
• Create tooling to accelerate root-cause analysis and reduce MTTR.
• Guide and mentor software engineering and SRE teams on monitoring best practices.
• Identify bottlenecks and single points of failure to improve reliability of the observability platform.
• Drive features from ideation to implementation and adoption.
• Collaborate on planning and refinement to ensure alignment and reduced complexity.
• Respond to alerts and provide on-call support for production issues.

Skills

Distributed systems

API design

Data pipelines

Software reliability

Go/Java

System design

Stakeholder management

Containerization

Terraform

Observability stack

Tools

Docker

Kubernetes

Terraform

Job description

This is the company

the company provides payments, data, and financial products in a single solution for customers like Meta, Uber, H&M, and Microsoft - making us the financial technology platform of choice. At the company, everything we do is engineered for ambition.

For our teams, we create an environment with opportunities for our people to succeed, backed by the culture and support to ensure they are enabled to truly own their careers. We are motivated individuals who tackle unique technical challenges at scale and solve them as a team. Together, we deliver innovative and ethical solutions that help businesses achieve their ambitions faster.

Staff Software Engineer - Platform Observability

the company provides a global, unified platform with a wide variety of financial services. Observability plays a key role providing comprehensive data to a wide variety of use cases, ensuring the platform’s reliability and supporting our operations to run smoothly.

As part of the Observability team you will build and maintain products and services that enable engineers at the company to understand how their services are behaving in real-time, reliably diagnose issues and streamline data discovery through a common observability ecosystem. Between logs, metrics and traces, our platform collects and processes billions of events per day.

What you will do:

As a Staff Software Engineer working on the observability of the platform, you will play a key role in shaping how teams work with telemetry data, driving strategic decisions, identifying and advocating for best practices for building, running and maintaining observable components.

This includes:

• Define and lead Logging, Metrics, Tracing & Alerting strategies
• Solve scaling bottlenecks in critical services in our telemetry data pipelines
• Create advanced tooling to accelerate root-cause analysis, reduce Mean Time to Resolution (MTTR), and eliminate alert fatigue
• Guide and mentor software engineering and SRE teams on monitoring best practices and instrumenting code.
• Identify and tackle bottlenecks and single points of failures in the architecture aiming to improve the reliability of the observability platform and the overall user experience.
• Driving features and products idealization, implementation and adoption.
• Collaborate on planning and refinement proactively providing input whilst striving for engineering alignment, quality and reducing complexity and dependencies.
• Respond to alerts and duty responsibilities, like providing support to other engineers and troubleshooting production issues

What you bring:

• You have 5+ years of relevant work experience with highly distributed systems
• Expertise in designing and implementing APIs and data pipelines for high-throughput, real-time data ingestion
• Experience improving software reliability between different categories including availability, performance, latency, efficiency, capacity, SLOs and incident management.
• Proficiency developing and maintaining software and frameworks written in Go and/or Java
• Advanced understanding of system design and evaluating its tradeoffs
• Strong stakeholder management, technical communication, and mentorship capability.
• Highly motivated to learn and continuously develop yourself
• Experience with containerisation and orchestration technologies (Docker, Kubernetes) and infrastructure as code tools(e.g: Terraform)
• Observability Stack Expertise: You have hands-on experience operating core telemetry data stores at scale e.g. Elasticsearch/Opensearch/VictoriaLogs/Clickhouse for logging, Prometheus/ VictoriaMetrics for metrics and Grafana Tempo for distributed tracing, Grafana LGTM stack, OpenTelemetry, Alertmanager, Clickhouse

Nice to haves:

• Experience with highly available/fault tolerant, replicated data storage systems, large scale data processing systems is a strong plus
• Infrastructure and Platform Experience
• Contributions to open-source observability projects

Our Diversity, Equity and Inclusion commitments

Our unique approach is a product of our diverse perspectives. This diversity of backgrounds and cultures is essential in helping us maintain our momentum. Our business and technical challenges are unique, and we need as many different voices as possible to join us in solving them - voices like yours. No matter who you are or where you’re from, we welcome you to be your true self at the company.

Studies show that women and members of underrepresented communities apply for jobs only if they meet 100% of the qualifications. Does this sound like you?

This role is based out of our Amsterdam office. We are an office-first company and value in-person collaboration; we do not offer remote-only roles.