Staff Software Engineer, Backend (Observability)

Affirm, Inc.; United States; Remote; USD 204,000 - 290,000; Full time

Benefits offered by this job

Remote-friendly work setup

100% subsidized medical coverage for你/

Job summary

Affirm, Inc. is seeking a Staff Software Engineer, Backend (Observability) to lead the telemetry platform across our services. This role blends hands-on engineering with roadmapping, mentoring six engineers, and partnering with infra and product teams.

You will define strategies to improve reliability, cost, and performance for 200+ TB/day of logs, metrics, and traces. You will design sampling, shaping observability for AI-assisted code and agentic development, and own the technical direction

Qualifications

• 8+ years of backend experience design, develop and launch at scale.
• Experience with distributed systems and telemetry pipelines.
• Strong knowledge of observability practices and SLOs.
• Experience with AWS, Kubernetes, and modern data tech.
• Excellent verbal and written communication for cross-functional teams.

Responsibilities

• Own the architecture of the observability telemetry pipeline for 200+ TB/day.
• Design log sampling strategies to reduce volume and cost while preserving signal.
• Shape how observability integrates with agentic development and AI tooling.
• Set technical strategy for the team on a yearly horizon and drive cross-team initiatives.
• Collaborate with product, design and analytics to ensure sustainability and manage risks.

Skills

Python

Kotlin

Distributed systems

Observability

Communication

Education

Bachelor's degree or equivalent practical experience

Tools

AWS

MySQL

Flink

Kubernetes

OpenTelemetry

Kafka

Terraform

Job description

Staff Software Engineer, Backend (Observability)

Remote US

At affirm, we exist for the moments that matter - giving people a clear, predictable way to pay over time, with no hidden fees, no surprises, and no tradeoffs on what matters most.

The Observability team builds the platform every affirm engineer relies on to understand the behavior and performance of our systems. Our platform ingests 200+ TB/day of metrics, logs, and traces from hundreds of services across 80+ Kubernetes clusters, and when a user's checkout fails, our data is how engineers find out why. Over the next year, we are moving to wide events, with one context-rich event per request, improving the signal-to-noise ratio to make debugging faster. We are also introducing log sampling and building observability best practices into agentic development and code review, so that AI-assisted code is well instrumented by default.

We're looking for a Staff Software Engineer to be the technical lead for Observability. You'll spend most of your time hands‑on while defining the roadmap, driving cross‑functional projects, mentoring a team of six engineers, and partnering closely with infrastructure and product teams.

What You'll Do

• You will own the architecture of affirm's telemetry pipeline, balancing reliability, query performance, and cost at 200+ TB/day.
• You will design log sampling strategies that significantly reduce volume and cost without losing the signal engineers need during incidents.
• You will shape how observability fits into agentic development, so that AI coding agents and code review catch instrumentation gaps before code ships.
• You will be responsible for setting technical strategy for your team on a year-long time scale, and help your team tie it together with critical, business‑impacting projects.
• You will collaborate across teams in the product development lifecycle by collaborating with product management, design & analytics to ensure technical sustainability, risks and trade‑offs are well understood and managed.
• You will act as a force‑multiplier for your team through your definition and advocacy of technical solutions and operational processes.
• You take ownership of your team's operations and availability by ensuring you have the right monitoring, triage rotations, playbooks, polcities, testing and alerting in place to support "keep the lights on" & on‑call efforts.
• You will foster a culture of quality and ownership on your team by setting code review and design standards for your team, and advocating for them beyond your team through your writing and tech talks.
• You will help develop talent on your team by providing feedback and guidance, and leading by example.

What We Look For

• You have 8+ years of experience designing, developing and launching backend systems at scale using languages like Python or Kotlin.
• You have an extensive track record of developing highly available distributed systems using technologies like AWS, MySQL, Flink and Kubernetes.
• You have built or operated high-volume telemetry systems (metrics, logs, and/or traces) using technologies like OpenTelemetry, Kafka, and Terraform.
• You have a well‑informed point of view on modern observability practices such as structured and wide events, sampling, cardinality management, and SLOs.
• You have driven adoption of a platform standard across many engineering teams, through both tooling and engagement.
• You have experience delivering major features, system components or deprecating existing functionality in a system through the definition of a technical and execution plan. You write high quality code that is easily understood and used by others.
• You thrive in ambiguity, and are comfortable moving from low level language idioms all the way to the architecture of large systems to understand how they work.
• Your growth and impact trajectory demonstrates that you have mastered gathering and iterating on feedback from your engineering and cross‑functional peers.
• You have strong verbal and written communication skills that support effective collaboration with our global engineering team.
• This position requires either equivalent practical experience or a Bachelor's degree in a related field.

Base Pay Grade - P

Equity Grade - 13

Employees new to affirm typically come in at the start of the pay range. affirm focuses on providing a simple and transparent pay structure which is based on a variety of factors, including location, experience and job‑related skills.

Base pay is part of a total compensation package that may include equity rewards, monthly stipends for health, wellness and tech spending, and benefits (including 100% subsidized medical coverage, dental and vision for you and your dependents.)

USA base pay range (CA, WA, NY, NJ, CT): $230,000 - $290,000

USA base pay range (all other U.S. states): $204,000 - $264,000

Remote‑first with flexibility built inaffirm is proud to be a remote‑first company. Most roles can be done from almost anywhere within the country of employment. Some positions may occasionally require in‑person work at an affirm office, and a few are office‑based due to the nature of the work. All new hires will be invited to attend an in‑person onboarding experience.

Benefits designed for youOur benefits reflect our commitment to care, transparency, and flexibility. Here are a few highlights:

• Health coverage at no cost: We cover 100% of premiums for employees and their dependents.
• Spending stipends: Monthly stipends support your tech setup, and the ability to choose health and wellness options that are right for you.
• Time off to recharge: Flexible time off and generous holiday calendars help you rest when you need to.
• Own a piece of what you build: Our employee stock purchase plan (ESPP) lets you buy affirm stock at a discount.

We're committed to providing an inclusive interview process, including accommodations for candidates with disabilities. If you need support, we're happy to help.

For positions based in San Francisco or Los Angeles: affirm considers qualified applicants with arrest and conviction records, as required by law.

Demographic Survey Statement (US‑based roles)

affirm is dedicated to building an inclusive culture where affirmers belong and have an equal opportunity to thrive.

To help us assess our outreach efforts, we invite candidates to voluntarily share information through a short demographic survey. This information is:

• Completely optional
• Anonymized and stored in aggregate
• Not accessible to recruiters or hiring managers at the individual level
• Never used in any hiring decision

Your decision to provide or withhold this information will not impact your candidacy in any way.

How do you identify? (gender identity) Select...

How do you identify? (race/ethnicity) Select...

Voluntary Self‑Identification

For government reporting purposes, we ask candidates to respond to the below self‑identification survey.Completion of the form is entirely voluntary. Whatever your decision, it will not be considered in the hiringprocess or thereafter. Any information that you do provide will be recorded and maintained in aconfidential file.

As set forth in affirm's Equal Employment Opportunity policy,we do not discriminate on the basis of any protected group status under any applicable law.

If you believe you belong to any of the categories of protected veterans listed below, please indicate by making the appropriate selection.As a government contractor subject to the Vietnam Era Veterans Readjustment Assistance Act (VEVRAA), we request this information in order to measurethe effectiveness of the outreach and positive recruitment efforts we undertake pursuant to VEVRAA. Classification of protected categoriesis as follows:

A "disabled veteran" is one of the following: a veteran of the U.S. military, ground, naval or air service who is entitled to compensation (or who but for the receipt of military retired pay would be entitled to compensation) under laws administered by the Secretary of Veterans Affairs; or a person who was discharged or released from active duty because of a service‑connected disability.

A "recently separated veteran" means any veteran during the three‑year period beginning on the date of such veteran's discharge or release from active duty in the U.S. military, ground, naval, or air service.

An "active duty wartime or campaign badge veteran" means a veteran who served on active duty in the U.S. military, ground, naval, or air service during a war, or in a campaign or expedition for which a campaign badge has been authorized under the laws administered by the Department of Defense.

An "Armed forces service medal veteran" means a veteran who, while serving on active duty in the U.S. military, ground, naval or air service, participated in a United States military operation for which an Armed Forces service medal was awarded pursuant to Executive Order 12985.