Principal Site Reliability Engineer

Tandem Diabetes Care; United States; Remote; USD 165,000 - 185,000; Full time

Benefits offered by this job

Health insurance

401(k) with company match

Employee Stock Purchase plan

Paid time off

Job summary

Tandem Diabetes Care is seeking a Principal Site Reliability Engineer to lead the reliability, availability, and performance of production systems. You will guide incident response, build runbooks, and advance IaC, CI/CD, and security readiness in collaboration with cross-functional teams.

This fully remote role within the United States emphasizes metrics-driven operations, on-call design, and mentoring of engineers across time zones, with a focus on reducing toil and maintaining regulatory

Qualifications

• Proven experience leading production support and incident management.
• Strong grounding in SRE: SLIs/SLOs, blameless postmortems, toil reduction.
• Owns on-call strategy, rotation design, alert tuning, and escalation.
• Experience with Terraform at scale and policy-as-code guardrails.
• Hands-on experience building CI/CD pipelines with reliability guardrails (GitHub Actions, Octopus Deploy, or Azure DevOps).

Responsibilities

• Lead production support and incident management end-to-end, including incident command and postmortems.
• Define and own SLOs/SLIs and drive monitoring and observability priorities.
• Own on-call strategy and alert tuning across time zones; build runbooks for rapid resolution.
• Develop and maintain infrastructure as code with Terraform; implement guardrails and policy code.
• Lead and mentor engineers, including remote/contract staff, to improve reliability practices.

Skills

Incident management

SRE fundamentals

On-call management

Leadership and mentoring

Education

B.S. in Computer Science

Cloud certifications (AWS/Azure/GCP)

Tools

Terraform

GitHub Actions

Octopus Deploy

Azure DevOps

Docker

Kubernetes

Prometheus

Grafana

Datadog

CloudWatch

Job description

Grow With Us

Grow With Us Tandem Diabetes Care creates new possibilities for people living with diabetes, their loved ones, and their healthcare providers through a positively different experience. We'd love for you to team up with us to \"innovate every day,\" put \"people first,\" and take the \"no-shortcuts\" approach that has propelled us to become a leader in the diabetes technology industry.

Stay Awesome

Stay Awesome Tandem Diabetes Care is proud to manufacture and sell the Tandem Mobi system and t:slim X2 insulin pump with Control-IQ+ technology --- an advanced predictive algorithm that automates insulin delivery. But we're so much more than that. Our company's human-centered approach to design, development, and support delivers innovative products and services for people who use insulin. Because many of our own team members live with diabetes, or have a loved one impacted by diabetes, the work is personal, and we are committed to the cause. Learn more at tandemdiabetes.com

A Day In The Life

The Principal Site Reliability Engineer (SRE) is responsible for the reliability, availability, and performance of the company's production systems. This role leads day-to-day production support and incident response and progressively replaces reactive firefighting with engineered SRE practice: SLOs, observability, on-call design, runbooks, and automation. It also advances infrastructure automation (Terraform/IaC), CI/CD reliability, disaster recovery, and security and compliance readiness in partnership with cross-functional teams. The role works with a distributed team that includes offshore consulting partners and is expected to raise their capability and independence; success is measured as much by what the team can do without the Principal SRE as by what the Principal SRE delivers personally.

The Principal Site Reliability Engineer (SRE)'s at Tandem are also responsible for:

Production Support & Incident Management

• Leads day-to-day production support: intake, triage, prioritization, escalation, queue health, and change execution.
• Establishes consistent support practices across a distributed team, including shift handoffs, ticket quality standards, and clear ownership of open issues.
• Leads incident management end-to-end: incident command, stakeholder communication, and blameless postmortems with corrective actions tracked to closure.
• Participates in and coordinates response activities for production security incidents, partnering with Security teams to contain threats, restore services, validate controls, and implement corrective actions.
• Owns on-call strategy: rotation design, escalation paths, alert tuning, and tooling (e.g., PagerDuty, New Relic), with explicit goals of reducing alert fatigue and building coverage that works across time zones. Participates as a senior escalation tier for high-severity incidents.
• Builds runbooks that standardize response to common failure modes and enable first-line resolution by engineers who did not build the system.

Reliability Engineering & Observability

• Defines and owns SLIs and SLOs for critical services, using them to guide monitoring, alerting, and reliability priorities.
• Drives systemic reduction of MTTD and MTTR through better instrumentation, alerting, diagnostics, and automation.
• Converts recurring support burden into permanent fixes, automation, or documentation rather than absorbing it as ongoing manual work.
• Establishes and maintains technology currency and lifecycle management practices for production platforms, ensuring cloud services, Kubernetes clusters, operating systems, runtimes, and infrastructure components remain supported, secure, and aligned with organizational standards. Proactively identifies and mitigates End-of-Life (EOL), End-of-Support (EOS), and technology obsolescence risks.
• Owns business continuity and disaster recovery readiness for production platforms, including backup and recovery strategies, recovery testing, failover capabilities, recovery runbooks, and adherence to defined RTO/RPO objectives.

Infrastructure Automation & CI/CD

• Leads infrastructure automation with Terraform, ensuring infrastructure is version-controlled, modular, reusable, and auditable, working within and improving existing patterns where they are sound.
• Eliminates toil through automation, reducing manual, repetitive operational work across the team.
• Adds reliability guardrails to CI/CD pipelines (automated rollback, change-risk checks, progressive delivery) with the teams that own them.

Operational Governance and Compliance

• Maintains production systems in accordance with applicable regulatory and compliance requirements, ensuring audit readiness and the ongoing effectiveness of access management, change management, logging, vulnerability remediation, patch management, and other operational controls.
• Maintains documentation and evidence supporting business continuity and disaster recovery controls, including recovery testing results, remediation plans, and audit artifacts.
• Partners with Security, Quality, and Compliance teams to maintain regulatory compliance, support internal and external audits, and drive timely resolution of audit findings and remediation activities.

Leadership, Mentorship & Collaboration

• Actively grows the capability of SRE and DevOps engineers, including consulting-partner engineers, through pairing, design and code review, and incident debriefs.
• Creates conditions in which junior and contract engineers raise concerns and disagree openly, recognizing that silent agreement is a production risk.
• Communicates in a documentation-first manner suited to a distributed, multi-time-zone team, so context is durable rather than held by one person.
• Partners with software engineering, QA, and architecture to embed reliability and operability into the development lifecycle rather than only after incidents.

Contributing Responsibilities (partners With Other Owners)

• Informs capacity planning and scaling strategy with the Test team (who own load testing and modeling) and introduces proactive resilience testing such as game days once baseline support and observability are stable.
• Partners with application, architecture, and business stakeholders to align business continuity and disaster recovery capabilities with application requirements and recovery objectives.
• Supports cloud cost optimization: rightsizing, reserved capacity, and observability spend governance.
• Ensures work complies with company policies, including Privacy/HIPAA and other regulatory, legal, and safety requirements. Other duties as assigned.

When & Where You'll Work

Remote: This position is fully remote and open to candidates within the United States. Equipment for the role will be provided and training will occur virtually.

What You'll Need

• We are looking for a forward-thinking professional who actively leverages AI to enhance, automate, and optimize internal tooling and system integrations. In this role, you won't just use existing systems---you will use AI to actively build smarter workflows and bridge gaps between our platforms.
• Demonstrated experience leading production support and incident management for production systems, including incident command during high-severity events.
• Strong grounding in SRE principles: SLIs/SLOs, blameless postmortems, toil reduction, and treating reliability as an engineering discipline rather than a purely operational function.
• Demonstrated experience owning on-call strategy, including rotation design, alert tuning, and escalation.
• Expertise with Terraform or comparable IaC at scale: module design, state management, and policy-as-code guardrails.
• Hands-on experience building CI/CD pipelines with reliability guardrails using tools such as GitHub Actions, Octopus Deploy, or Azure DevOps.
• Deep experience with at least one major cloud platform (AWS, Azure, or GCP; [ preferred]) and with containerization and orchestration (Docker/Kubernetes).
• Working knowledge of observability tooling (e.g., Prometheus, Grafana, Datadog, CloudWatch, ELK/OpenSearch) and using it to drive SLO-based alerting.
• Experience designing and testing disaster recovery, including backup/restore, failover, and RTO/RPO validation.
• Working knowledge of cloud security and compliance practices (IAM, network segmentation, encryption, vulnerability and patch management) and cloud cost optimization.
• Proficiency in at least one scripting or programming language (e.g., Python, Go, Bash) for automation and tooling.
• Experience in FDA and ISO regulated industries and with agile methodologies preferred.

Extra Awesome

• B.S. in Computer Science or an equivalent combination of education and applicable job experience, including technical school training and certifications in networks, servers, and cloud infrastructure; demonstrated production experience weighs more heavily than degree.
• Relevant cloud certifications (e.g., AWS/Azure/GCP Professional or Architect level) preferred.
• 10+ years in Site Reliability Engineering, DevOps, or infrastructure engineering, including:
• Leading production support, incident management, and postmortem practice.
• Managing production cloud infrastructure as code (Terraform or equivalent) at scale.
• Operating and improving CI/CD pipelines in a production environment.
• Designing, executing, and testing disaster recovery plans.
• Partnering with security teams on production security and compliance posture.
• 2+ years mentoring or technically leading other engineers, including engineers who are remote, offshore, or contracted through a partner organization.

Compensation & Benefits

The starting base pay range for this position is**$165,000 to $185,000** annually. Base pay will vary based on job-related knowledge, skills, experience and may also fluctuate depending on candidate's location and the overall job market. In addition to base pay, Tandem offers a competitive compensation package that includes bonus and a robust benefits package.

Tandem offers health care benefits such as medical, dental, vision available your first day, as well as health savings accounts and flexible saving accounts. You'll also receive 11 paid holidays per year, a minimum of 20 days of paid time off (with accrual starting on day 1) and you will have access to a 401k plan with company match as well as an Employee Stock Purchase plan. Learn more about Tandem's benefits here!

You Should Know

Qualified applicants with arrest or conviction records will be considered for employment in accordance with applicable state and local Fair Chance laws and regulations. A conditional offer of employment from Tandem is contingent upon successful completion of a pre-employment screening process comprised of a drug test (excluding marijuana) and background check, which includes a review of criminal history information.

Tandem has good cause to conduct a review of criminal history information of candidates for this position, as this role may involve access to proprietary, sensitive and/or confidential information, including customer protected health information. This review is required to ensure that individuals in such roles uphold high standards of trust and integrity so as to protect the interests of our customers, employees, and stakeholders.

Why You'll Love Working Here

At Tandem, we believe joy fuels excellence. That's why we've built a workplace that celebrates your achievements and supports your well-being. Our team thrives on pushing boundaries and fostering growth, all while maintaining a spirit of fun and camaraderie. This is just one of the ways we stay awesome! Explore the benefits and reasons to love Tandem at https://www.tandemdiabetes.com/careers.

BE YOU, WITH US!

We embrace the value that every single one of us brings to the table. But sometimes we forget that when we don't meet 100% of a job description's criteria -- maybe you're feeling that way right now? We want you to be you, with us.

Tandem is firmly committed to being an equal opportunity employer and does not discriminate on the basis of age, disability, sex, race, religion or belief, gender identity or expression, marriage/civil partnership, pregnancy/maternity, or sexual orientation. We are an inclusive organization, and we welcome applications from a wide range of candidates. Selection for roles will be based on individual merit alone.

Referrals

We love a good referral! If you know someone who would be a great fit for this position, please share!

Application Deadline

The position will be posted until a final candidate is selected for the requisition or the requisition has a sufficient number of applications.

Make a move that matters. Join Tandem Diabetes Care, where we're turning challenges into triumphs every day and where your talents will help shape a healthier, happier tomorrow.