Staff Backend Observability Lead (Remote)

Affirm, Inc.; Madison (WI); Remote; USD 204,000 - 264,000; Full time

Benefits offered by this job

Remote-first with flexibility

Equity opportunities

Healthcare coverage for you and depend

Stipends for tech setup

Job summary

AFFIRM is seeking a Staff Software Engineer for Observability to lead the telemetry platform that powers the company’s systems. You’ll blend hands-on coding with long‑term roadmapping, mentoring six engineers and partnering with infrastructure and product teams.

You’ll own the telemetry architecture, design sampling, and shape how observability integrates with agentic development, while driving strategic initiatives that improve reliability and efficiency.

Qualifications

• 8+ years of experience designing, developing and launching backend systems at scale using languages like Python or Kotlin.
• Extensive track record of developing highly available distributed systems using AWS, MySQL, Flink and Kubernetes.
• Built or operated telemetry systems (metrics, logs, traces) using OpenTelemetry, Kafka, Terraform.
• Well-informed on modern observability practices such as wide events, sampling, cardinality, and SLOs.
• Driven adoption of a platform standard across engineering teams through tooling and engagement.

Responsibilities

• Own architecture of AFFIRM’s telemetry pipeline for 200+ TB/day, balancing reliability, performance and cost.
• Design log sampling strategies to reduce volume and cost without losing signal during incidents.
• Shape how observability fits into agentic development with AI coding agents and code reviews.
• Set technical strategy for the team on a year-long horizon and align with business initiatives.
• Collaborate with product management, design and analytics to ensure sustainability and risk management.
• Serve as a force-multiplier for the team through technical solutions and operational processes.
• Own team operations and availability with monitoring, on-call playbooks, testing and alerts.
• Foster quality and ownership by setting code review and design standards company-wide.
• Mentor and develop team members through feedback and leading by example.

Skills

Python

Kotlin

AWS

MySQL

Flink

Kubernetes

OpenTelemetry

Kafka

Terraform

Education

Bachelor’s degree or equivalent

Tools

AWS

MySQL

Flink

Kubernetes

OpenTelemetry

Kafka

Terraform

Job description

AFFIRM is seeking a Staff Software Engineer for Observability to lead the telemetry platform that powers the company’s systems. You’ll blend hands-on coding with long‑term roadmapping, mentoring six engineers and partnering with infrastructure and product teams.

You’ll own the telemetry architecture, design sampling, and shape how observability integrates with agentic development, while driving strategic initiatives that improve reliability and efficiency.