Staff Backend Engineer, Observability Lead (Remote)

Affirm, Inc.; Washington (District of Columbia); Remote; USD 204,000 - 290,000; Full time

Benefits offered by this job

Health coverage

Monthly stipends for tech setup

Flexible time off

Employee stock purchase plan

Job summary

AFFIRM is seeking a Staff Software Engineer to lead Observability. You’ll own the telemetry pipeline architecture, drive log sampling, and guide cross-functional teams in building scalable, observable systems.

You will mentor six engineers, shape technical strategy for the year, and collaborate with product, design and analytics to ensure robust monitoring and incident response. Remote-friendly, with strong engineering impact.

Qualifications

• 8+ years of experience designing, developing and launching backend systems at scale using languages like Python or Kotlin.
• Extensive track record of developing highly available distributed systems using AWS, MySQL, Flink and Kubernetes.
• Built or operated high-volume telemetry systems (metrics, logs, and/or traces) using OpenTelemetry, Kafka, and Terraform.

Responsibilities

• Own the architecture of AFFIRM’s telemetry pipeline, balancing reliability, query performance, and cost at 200+ TB/day.
• Design log sampling strategies that significantly reduce volume and cost without losing the signal engineers need during incidents.
• Shape how observability fits into agentic development, so that AI coding agents and code review catch instrumentation gaps before code ships.
• Set technical strategy for your team on a year-long time scale, and align it with business-impacting projects.
• Collaborate across teams in the product development lifecycle with product management, design & analytics to ensure sustainability, risks, and trade-offs are understood.
• Act as a force-multiplier for your team through the definition and advocacy of technical solutions and operational processes.
• Own your team’s operations and availability by ensuring monitoring, triage rotations, playbooks, policies, testing and alerting are in place to support on-call efforts.
• Foster a culture of quality and ownership by setting code review and design standards and promoting them beyond your team.
• Develop talent by providing feedback, guidance, and leading by example.

Skills

Python

Kotlin

Distributed systems

AWS

Kubernetes

OpenTelemetry

Kafka

Terraform

Log sampling

Education

Bachelor’s degree or equivalent

Tools

OpenTelemetry

Kafka

Terraform

MySQL

Flink

Kubernetes

AWS

Job description

AFFIRM is seeking a Staff Software Engineer to lead Observability. You’ll own the telemetry pipeline architecture, drive log sampling, and guide cross-functional teams in building scalable, observable systems.

You will mentor six engineers, shape technical strategy for the year, and collaborate with product, design and analytics to ensure robust monitoring and incident response. Remote-friendly, with strong engineering impact.