Staff Software Engineer , Observability

Logoslabs; San Francisco (CA); Remote; USD 180,000 - 270,000; Full time

Benefits offered by this job

Comprehensive Healthcare Benefits and

401k with Employer Match

Global Benefit programs

Job summary

Reddit’s Observability team builds and maintains the platform that powers monitoring, logging, and tracing for billions of events. You will work with open source tools (Prometheus, Thanos, Grafana, Vector, Clickhouse, Otel, Loki) to design scalable systems that help engineers understand deployments at scale.

The role emphasizes collaboration, reliability, and performance across a distributed infrastructure, with on-call responsibilities and opportunities to contribute to upstream projects.

Qualifications

• 7+ years in internet-scale software development, infrastructure focus preferred.
• Strong background in distributed systems and observability tooling.
• Experience with Kubernetes-based platforms and operators is a plus.
• Excellent communication and collaboration across teams.

Responsibilities

• Collaborate with software engineers to build the foundational observability platform.
• Deliver software to improve availability, scalability, latency, and efficiency of observability components.
• Contribute upstream changes to open source projects we use.
• Share on-call responsibilities and automate critical processes.

Skills

Internet-scale software development

Distributed systems

Kubernetes experience

Troubleshooting

Communication

Tools

Prometheus

Thanos

Grafana

Vector

Clickhouse

OTel

Loki

Kubernetes

Job description

Reddit is a community of communities. It’s built on shared interests, passion, and trust, and is home to the most open and authentic conversations on the internet. Every day, Reddit users submit, vote, and comment on the topics they care most about. With 100,000+ active communities and approximately 130 million daily active unique visitors, Reddit is one of the internet’s largest sources of information. For more information, visit www.redditinc.com.

This role is fully remote. Reddit has a flexible first workforce.

The Observability (OBS) team is looking to hire an Engineer that thrives at the intersection of infrastructure and software development. This team owns a suite of tools for allowing engineers to understand their creations, based primarily on open-source solutions at scale. We’re active users of and contributors to our open source tools, including Prometheus, Thanos, and more.

Monitoring

Our monitoring stack at Reddit processes billions of data-points a minute, delivering insights at scale. Reddit has one of the larger deployments of Prometheus and Thanos in the world , and with this come unique challenges of scale for these systems. Fun problems include performance engineering on a distributed query system and product thinking around new features to remove the user pain from this stack.

Logging

We operate a platform for logging based on Vector and Clickhouse that processes millions of logs a day for Reddit. Thousands of people at Reddit use this system every day as they maintain, deploy, and debug their code. Join us as we invest in this system to help users better understand the data they have.

Distributed Tracing

Tracing at Reddit is based on OTEL, Clickhouse, and Grafana. It’s light on features, but currently processing tens of millions of events, and we will be building new ways for users to leverage and understand this data in the coming year.

As a member of the Observability team, your work will span these domains, which are rich with challenging infrastructure and software engineering problems. Your work will directly impact hundreds of millions of users around the world. Join us and help build the future of Reddit!

In your day-to-day, you can expect to:

• Work collaboratively with a team of software engineers to create and maintain the foundational platform for running Reddit’s infrastructure.

• Deliver software to improve the availability, scalability, latency, and efficiency of observability components.

• Contribute feedback to the technical and strategic direction of eventing at Reddit.

• Automate critical aspects of the event driven development process

• Share on-call responsibilities.

• Contribute upstream changes to the open source projects we use

You have:

• 7+ years of experience developing internet-scale software, preferable in the context of infrastructure.

• Familiarity with distributed systems development, bonus if familiar with any of the specific tools (Prometheus, Thanos, Grafana, Vector, Clickhouse, Otel, Loki)

• Experience developing on top of Kubernetes or similar distributed systems.

  • Kubernetes controller or operator development experience is a huge plus.

• Strong troubleshooting capabilities surrounding both systems and software.

• Experience engineering large systems, tracking work, and being a self-starter on projects.

• Excellent communication skills to collaborate with a service-oriented team and company.

Benefits:

• Comprehensive Healthcare Benefits and Income Replacement Programs

• 401k with Employer Match

• Global Benefit programs that fit your lifestyle, from workspace to professional development to caregiving support

• Family Planning Support

• Gender-Affirming Care

• Mental Health & Coaching Benefits

• Flexible Vacation & Paid Volunteer Time Off

• Generous Paid Parental Leave