Senior Site Reliability Engineer - Storage

Socket.dev; San Francisco (CA); On-site; USD 140,000 - 210,000; Full time

Benefits offered by this job

Health coverage

Dental coverage

Vision coverage

401k with match

Wellness stipend

Commuter stipend

Job summary

Lambda is building the world's best AI cloud. We seek a Storage Engineer to own reliability and capacity for our production storage fleet, spanning on-site data centers and a software-defined data plane. You will automate, monitor, and optimize performance across multiple sites, collaborating with hardware, networking, and release teams.

You will drive incident response, build dashboards, and implement self-healing automation while advancing CI/CD for storage tooling and integrations.

Qualifications

• 5+ years operating Linux systems in production or HPC environments.
• Hands-on storage experience at scale on software-defined platforms.
• Strong incident-response ownership from alert to postmortem.
• Experience with monitoring/logging: Prometheus, Grafana, Datadog, Sumo Logic.
• Kubernetes experience including GitOps tools like ArgoCD, Helm, Kustomize.
• CI/CD tooling: GitHub Actions, Jenkins, BuildKite; Docker/Podman.
• Programming in Python or Go; Infrastructure as Code with Terraform/Ansible.
• Core storage protocols: NFS, SMB, S3, NVMe-oF; basic DB concept.

Responsibilities

• Own reliability, performance, and capacity health of production storage fleet.
• Build and maintain monitoring, dashboards, and alerting for storage systems.
• Investigate and resolve storage incidents using telemetry and profiling.
• Automate ticketing, escalation, and incident-response workflows.
• Design self-healing automation for drive replacements, node swaps, rebuilds.
• Implement CI/CD pipelines for storage automation and tooling.
• Collaborate with Storage Engineers, Fleet Orchestration, Release Engineering.
• Work with hardware and networking teams to diagnose low-level I/O issues.
• Participate in on-call rotations to minimize MTTR.

Skills

Linux production systems

Storage at scale (SDS)

Incident response

Monitoring dashboards

Kubernetes

CI/CD tooling

Python/Go

Infrastructure as code

Storage protocols

Education

Bachelor's degree in Computer Science or related field

Tools

Prometheus

Grafana

Alertmanager

Datadog

Sumo Logic

Kubernetes

ArgoCD

Helm

Kustomize

GitHub Actions

Jenkins

BuildKite

Docker

Terraform

Ansible

Job description

Lambda, The Superintelligence Cloud, is a leader in AI cloud infrastructure serving tens of thousands of customers. Our customers range from AI researchers to enterprises and hyperscalers. Lambda's mission is to make compute as ubiquitous as electricity and give everyone the power of superintelligence. One person, one GPU.

If you'd like to build the world's best AI cloud, join us.

*Note: This position requires presence in our San Francisco or San Jose office location 4 days per week; Lambda’s designated work from home day is currently Tuesday.

Lambda's Storage Engineering team is the backbone behind our world-class storage offerings, operating at a scale seldom seen in today's markets. We own the full spectrum of Lambda's data platform services—from low-level storage systems to the APIs and tooling our customers build on every day.

Our work directly powers some of the most demanding compute workloads in the industry, which means reliability and performance aren't just goals—they're the baseline. We're looking for engineers who want to solve problems at scale, own critical systems end to end, and help shape the future of infrastructure for AI and beyond

What You’ll Do

• Own the reliability, performance, and capacity health of Lambda's production storage fleet across all data centers, operating behind Lambda's own software-defined data plane.

• Build and maintain monitoring, dashboards, and alerting for storage performance, capacity, and hardware failures.

• Investigate and resolve storage-related incidents using deep telemetry, logs, and performance profiling — from a single flapping NIC to a cluster-wide rebuild.

• Automate ticketing, escalation, and incident-response workflows so the team spends less time on repetitive triage and more time on root cause.

• Design and maintain self-healing automation for common failure modes: drive replacement, node swaps, rebuild monitoring, and capacity rebalancing.

• Implement CI/CD pipelines for storage automation and tooling.

• Partner with Storage Engineers, Fleet Orchestration, and Release Engineering to automate the deployment and configuration of software‑defined storage across new and existing sites using tools such as Ansible, Jenkins etc.

• Work with hardware and networking teams to diagnose low-level I/O and network issues — NIC errors, RDMA/RoCE/InfiniBand fabric health, path multipathing — that surface as storage‑layer symptoms.

• Participate in an on‑call rotation supporting Lambda's storage fleet, with a focus on driving down MTTR and building the automation that keeps you from getting paged for the same thing twice.

You Have

• 5+ years of experience operating Linux systems in production or HPC environments, with hands‑on storage experience at scale on scale‑out or software‑defined platforms (e.g., CEPH, Lustre, GPFS, or similar).

• Hands‑on experience operating Software‑Defined Storage (SDS) platforms at scale, including integrating with their management and data‑plane APIs.

• Strong incident‑response instincts: comfortable owning a production storage incident end to end, from first alert through root cause to postmortem.

• Working experience with monitoring and logging platforms such as Prometheus, Grafana, Alertmanager, Datadog, or SumoLogic — including building dashboards and alert/pager routing for multiple audiences.

• Working experience with Kubernetes (GitOps tooling such as ArgoCD, Helm/Kustomize) and hands‑on troubleshooting.

• Working experience with CI/CD tooling (GitHub Actions, Jenkins, BuildKite), containerization (Docker/Podman), and systems programming in Python or Go.

• Working experience with Infrastructure as Code (Terraform, Ansible).

• Solid understanding of core storage protocols across file (NFS, SMB), object (S3), block (NVMe-oF/TCP), and structured (vector DB, SQL) storage.

Nice to Have

• Experience with cutting‑edge software‑defined storage solutions such as VAST or Weka.

• Enterprise storage expertise in solutions such as NetApp, Dell PowerScale, GPFS, or Lustre.

• Experience writing or operating Kubernetes CSI drivers.

• Experience with SR‑IOV and virtualization (KVM/QEMU).

• Experience with GPUDirect Storage, RDMA, InfiniBand, or RoCE networking.

• Familiarity with NIC‑level diagnostics (ethtool, mlxlink) and fleet‑wide operations tooling (clush or similar).

• Contributions to open‑source storage projects.

Salary Range Information

The annual salary range for this position has been set based on market data and other factors. However, a salary higher or lower than this range may be appropriate for a candidate whose qualifications differ meaningfully from those listed in the job description.

About Lambda

• Founded in 2012, with 500+ employees, and growing fast

• Our investors notably include TWG Global, US Innovative Technology Fund (USIT), Andra Capital, SGW, Andrej Karpathy, ARK Invest, Fincadia Advisors, G Squared, In‑Q‑Tel (IQT), KHK & Partners, NVIDIA, Pegatron, Supermicro, Wistron, Wiwynn, Gradient Ventures, Mercato Partners, SVB, 1517, and Crescent Cove

• We have research papers accepted at top machine learning and graphics conferences, including NeurIPS, ICCV, SIGGRAPH, and TOG

• Our values are publicly available: https://lambda.ai/careers

• We offer generous cash & equity compensation

• Health, dental, and vision coverage for you and your dependents

• Wellness and commuter stipends for select roles

• 401k Plan with 2% company match (USA employees)

• Flexible paid time off plan that we all actually use

Equal Opportunity Employer

Lambda is an Equal Opportunity employer. Applicants are considered without regard to race, color, religion, creed, national origin, age, sex, gender, marital status, sexual orientation and identity, genetic information, veteran status, citizenship, or any other factors prohibited by local, state, or federal law.