Staff Backend Engineer, Observability Lead - Remote

Affirm, Inc.; Houston (TX); Remote; USD 204,000 - 290,000; Full time

Benefits offered by this job

Health coverage for you and dependents

Monthly stipends for tech setup

Flexible time off and holidays

ESPP – employee stock purchase plan

Job summary

AFFIRM is seeking a Staff Software Engineer to lead Observability. You’ll own the telemetry pipeline architecture, balance reliability, performance, and cost, and guide cross‑functional projects with product, design, and analytics. You’ll mentor six engineers and help set the team’s strategic direction.

You’ll design log sampling, shape observability practices, and champion quality through code standards and reviews across the engineering organization.

Qualifications

• You have 8+ years of experience designing, developing and launching backend systems at scale using languages like Python or Kotlin.
• You have an extensive track record of developing highly available distributed systems using technologies like AWS, MySQL, Flink and Kubernetes.
• You have built or operated high-volume telemetry systems (metrics, logs, and/or traces) using technologies like OpenTelemetry, Kafka, and Terraform.
• You have a well-informed point of view on modern observability practices such as structured and wide events, sampling, cardinality management, and SLOs.
• You have driven adoption of a platform standard across many engineering teams, through both tooling and engagement.
• You have experience delivering major features, system components or deprecating existing functionality in a system through the definition of a technical and execution plan. You write high quality code that is easily understood and used by others.
• You thrive in ambiguity, and are comfortable moving from low level language idioms all the way to the architecture of large systems to understand how they work.
• Your growth and impact trajectory demonstrates that you have mastered gathering and iterating on feedback from your engineering and cross‑functional peers.
• You have strong verbal and written communication skills that support effective collaboration with our global engineering team.
• This position requires either equivalent practical experience or a Bachelor’s degree in a related field.

Responsibilities

• You will own the architecture of AFFIRM’s telemetry pipeline, balancing reliability, query performance, and cost at 200+ TB/day.
• You will design log sampling strategies that significantly reduce volume and cost without losing the signal engineers need during incidents.
• You will shape how observability fits into agentic development, so that AI coding agents and code review catch instrumentation gaps before code ships.
• You will be responsible for setting technical strategy for your team on a year-long time scale, and help your team tie it together with critical, business-impacting projects.
• You will collaborate across teams in the product development lifecycle by collaborating with product management, design & analytics to ensure technical sustainability, risks and trade-offs are well understood and managed.
• You will act as a force‑multiplier for your team through your definition and advocacy of technical solutions and operational processes.
• You take ownership of your team’s operations and availability by ensuring you have the right monitoring, triage rotations, playbooks, polcities, testing and alerting in place to support “keep the lights on” & on‑call efforts.
• You will foster a culture of quality and ownership on your team by setting code review and design standards for your team, and advocating for them beyond your team through your writing and tech talks.
• You will help develop talent on your team by providing feedback and guidance, and leading by example.

Skills

Backend system design

Python

Kotlin

Distributed systems

AWS

MySQL

Flink

Kubernetes

OpenTelemetry

Kafka

Terraform

Observability

Technical leadership

Cross-functional collaboration

Code quality

Education

Bachelor's degree

Tools

AWS

MySQL

Flink

Kubernetes

OpenTelemetry

Kafka

Terraform

Job description

AFFIRM is seeking a Staff Software Engineer to lead Observability. You’ll own the telemetry pipeline architecture, balance reliability, performance, and cost, and guide cross‑functional projects with product, design, and analytics. You’ll mentor six engineers and help set the team’s strategic direction.

You’ll design log sampling, shape observability practices, and champion quality through code standards and reviews across the engineering organization.