Senior Director, Network Reliability Engineering

Lambda Inc.; Bellevue (WA); Hybrid; USD 250,000 - 350,000; Full time

Benefits offered by this job

Health, dental, vision coverage

Wellness stipend

Commuter stipend

401k with company match

Flexible paid time off

Job summary

Lambda, The Superintelligence Cloud, is seeking a Senior Director of Network Reliability Engineering to own the health of our global network end to end. You will lead network SRE, automation engineering, incident management, and break-fix coordination across data centers and edge locations.

You will hire and grow a leaders-driven team, design self-healing architectures, and partner with product to define reliability targets.

Qualifications

• 8+ years of engineering management experience, including managing managers.
• Proven track record leading SRE or production operations for large-scale cloud networks.
• Background in network or software engineering; able to discuss architecture and tradeoffs.
• Experience reducing toil through automation with measurable results.
• Familiar with routing, leaf-spine fabrics, optics, and network telemetry.

Responsibilities

• Own the reliability of Lambda's network end to end across data centers and edge.
• Lead break-fix operations, fault detection, triage and hardware repair coordination.
• Manage on-call, incident command, escalation, and blameless postmortems.
• Define multi-year strategy for a self-healing network with automated loops.
• Convert manual break-fix work into durable software and reduce toil.
• Define network SLOs and error budgets with product teams.
• Build safe automation with blast-radius controls and rollbacks.
• Hire, develop and retain engineers; grow the organization as the network scales.
• Collaborate with Data Center Ops, Capacity Automation and vendors.
• Represent the org to senior leadership and customers during incidents.

Skills

Engineering management

SRE leadership

Network reliability

Automation & tooling

Incident management

Routing (BGP)

Network telemetry

Closed-loop remediation

Data-driven automation

Team leadership

Business goal translation

Decision making under pressure

Communication (written & verbal)

Job description

Lambda, The Superintelligence Cloud, is a leader in AI cloud infrastructure serving tens of thousands of customers. Our customers range from AI researchers to enterprises and hyperscalers. Lambda's mission is to make compute as ubiquitous as electricity and give everyone the power of superintelligence. One person, one GPU.

If you'd like to build the world's best AI cloud, join us.

*Note: This position requires presence in our Bellevue or San Francisco office location 4 days per week; Lambda’s designated work from home day is currently Tuesday.

Lambda, The Superintelligence Cloud, is a leader in AI cloud infrastructure serving tens of thousands of customers. Our customers range from AI researchers to enterprises and hyperscalers. Lambda's mission is to make compute as ubiquitous as electricity and give everyone the power of superintelligence. One person, one GPU.

Every customer workload on Lambda depends on the network, from a single-GPU experiment to a training run spanning thousands of GPUs. When an optic degrades, a link flaps or a switch fails, customers feel it as stalled jobs, restarts and lost compute time. We will operate the existing network flawlessly for all customer workloads while we reinvent how failures are found and fixed, and the automation underneath all of it.

Lambda has 10x'd over the last three years, and the network engineering organization is scaling to match. We're looking for a Senior Director of Network Reliability Engineering to own the health of that network end to end: the break-fix operations that restore service today, and the software that detects, isolates and repairs failures automatically so that most never reach a human or a customer.

You'll lead an organization spanning network SRE and automation engineering, incident management, and break-fix coordination with our data center teams. We're looking for an experienced leader of leaders who treats every repeated failure as a software problem, measures reliability and builds toward a self-healing network where detection, diagnosis and remediation run as automated closed loops. You'll hire and grow a talented team of software and network proficient engineers, shape team culture, and partner with technical leadership to keep reliability ahead of our scale ambitions.

This role will report to the VP of Cloud and AI Networking, working alongside a highly capable and passionate team of engineers. If you'd like to join a great team of customer obsessed engineers building the world's best AI cloud, come join us.

What You'll Do

• Own the reliability of Lambda's network end to end, from GPU compute fabrics to data center, backbone and edge, and be accountable for availability, time to detect and time to restore

• Run break-fix operations at scale: fault detection, triage, hardware replacement and RMA, optics and cabling remediation, and coordination with Data Center Operations and remote-hands partners across a growing fleet of data centers

• Lead the networking ops excellence and incident management program, including on-call, incident command, escalation paths, customer communication and blameless postmortems that turn into engineering fixes

• Set the multi-year strategy for a self-healing network: automated loops that detect and isolate failing components, shift traffic, drain and restore capacity, and open repair work without human intervention

• Convert manual break-fix work into durable software. The goal is a network that repairs itself, not a faster set of runbooks. Measure toil, set targets to reduce it, and track the share of failures resolved automatically as a primary outcome for your org

• Define network SLOs and error budgets with product and customer-facing teams, and use them to prioritize reliability work against feature and capacity work

• Build automation that is safe to trust, with blast-radius limits, staged rollouts, audit trails and rollback for every automated action, including AI-assisted triage and remediation

• Own the operational bar for your org: how incidents are detected, handled, reviewed, measured and learned from

• Build and lead the organization that delivers it: hire, develop and retain engineers and the leaders who manage them, and design the team structure as the network and the org grow

• Manage through leads and senior ICs, developing both, with regular 1:1s, clear performance feedback, growth planning, and sponsorship of meaningful work

• Partner with Network Engineering, Network Capacity Automation, Software and Principal Engineers so reliability requirements are designed in from the start and failure data informs architecture, hardware and vendor decisions

• Partner with Data Center Operations, Supply Chain and vendors on spares strategy, RMA turnaround and hardware failure trends

• Represent your org's work, reliability metrics and technical direction to senior leadership, and to enterprise customers and partners during major incidents and reviews

• Contribute to org-wide engineering process improvements: how we plan, how we ship, how we learn from incidents

You

• Have 8+ years of engineering management experience, including managing managers and directly managing senior individual contributors

• Proven track record of leading SRE, network reliability or production operations organizations for large-scale cloud or data center networks with 24x7 availability requirements

• Have a network or software engineering background. You've shipped production systems and can engage credibly on architecture, technical tradeoffs, and code quality

• Have measurably reduced toil and incident impact through automation, and can show the data, such as lower time to restore or a higher share of incidents remediated automatically

• Familiarity designing, building, scaling and operating large-scale networks, including routing (BGP), leaf-spine fabrics, optics and hardware failure modes, and network telemetry and monitoring

• Experience building closed-loop or event-driven remediation systems, and the safety controls that keep automation from causing incidents

• Have run incident management at scale: incident command, a blameless postmortem culture, and turning lessons into engineering work that ships

• Foster a data driven, automation first, high velocity organization

• Have a track record of building high-performing teams in fast-moving, technically demanding environments

• Are skilled at translating ambiguous business and product goals into clear team priorities and executable engineering plans

• Show strong judgment about when to go deep technically, when to delegate, and when to escape

• Stay calm and decisive during major incidents, and communicate clearly with customers and executives while one is in progress

• Excellent communicator both written and verbal. You use high quality written artifacts to inform high quality decision making and get approval on problems and solution opportunities

Nice to Have

• Experience in GPU cloud, HPC, or AI/ML infrastructure environments

• Experience operating InfiniBand or RoCE fabrics at scale, and understanding how network faults affect distributed training jobs

• Experience applying ML or LLM-based tooling to operations, such as anomaly detection, alert correlation, automated triage or root-cause analysis

• Experience with streaming network telemetry at scale (gNMI, OpenConfig or similar)
  

Salary Range Information

The annual salary range for this position has been set based on market data and other factors. However, a salary higher or lower than this range may be appropriate for a candidate whose qualifications differ meaningfully from those listed in the job description.

About Lambda

• Founded in 2012, with 500+ employees, and growing fast

• Our investors notably include TWG Global, US Innovative Technology Fund (USIT), Andra Capital, SGW, Andrej Karpathy, ARK Invest, Fincadia Advisors, G Squared, In-Q-Tel (IQT), KHK & Partners, NVIDIA, Pegatron, Supermicro, Wistron, Wiwynn, Gradient Ventures, Mercato Partners, SVB, 1517, and Crescent Cove

• We have research papers accepted at top machine learning and graphics conferences, including NeurIPS, ICCV, SIGGRAPH, and TOG

• Our values are publicly available: https://lambda.ai/careers

• We offer generous cash & equity compensation

• Health, dental, and vision coverage for you and your dependents

• Wellness and commuter stipends for select roles

• 401k Plan with 2% company match (USA employees)

• Flexible paid time off plan that we all actually use

Equal Opportunity Employer

Lambda is an Equal Opportunity employer. Applicants are considered without regard to race, color, religion, creed, national origin, age, sex, gender, marital status, sexual orientation and identity, genetic information, veteran status, citizenship, or any other factors prohibited by local, state, or federal law.