Software Engineer, Infrastructure Services (Cloud Network Fault Tolerance)

Socket.dev; Cupertino (CA); On-site; USD 210,000 - 280,000; Full time

Job summary

Apple IS&T Infrastructure Services is seeking engineers to help build fault-tolerant, large-scale distributed systems that manage Apple's hyper-scale cloud network. You will design sensing, decision-making, and remediation layers to detect and fix failures automatically with minimal human intervention.

Applicants should have deep fundamentals in distributed systems, graph algorithms, and experience with chaos engineering.

Qualifications

• Bachelor’s degree or equivalent practical experience.
• Strong fundamentals in distributed systems and graph algorithms.
• Proficiency in Go, C++, Rust, or Python.
• Experience with fault-tolerant mechanisms and automated self-healing.
• Chaos engineering and fault injection in prod or staging.
• Ownership: design docs, code reviews, and post‑mortem analyses.

Responsibilities

• Build fault-tolerant distributed systems for hyper‑scale networks.
• Design sensing, decision‑making, and remediation layers for automatic failure handling.
• Automate safe remediation actions in production without introducing new risk.
• Translate research ideas from papers into robust production systems.

Skills

Distributed systems

Graph algorithms

Go/C++/Rust/Python

Fault-tolerant design

Chaos engineering

Post-mortem analyses

Education

Bachelor's degree or equivalent

Master's or Ph.D. preferred

Tools

eBPF

XDP

OVS

Job description

Do you want to help build some of the largest and most consequential enterprise and customer technology systems in the world? Join Apple’s Information Systems and Technology (IS&T) organization. IS&T is the engine behind everything Apple does for customers and the people who build for them. It’s Apple’s central nervous system. Supporting 2.5 billion active Apple devices, processing billions of secure transactions, and keeping the technology that defines modern life running flawlessly, IS&T makes the impossible feel effortless.” Do you love building solutions to handle global complexity and immense scale? Imagine what you could do here. Infrastructure Services is part of IS&T and the foundation of Apple's global network operations — managing data center equipment and systems to deliver compute, storage, and networking services for teams across Apple, including its internal developer community. From individual facilities to a worldwide network, Infrastructure Services ensures the technology underneath everything works without question. This is not a traditional network engineering role. This is a software engineering role for builders who love deep technical challenges, have strong fundamentals in distributed systems and fault-tolerant design, and want to turn cutting-edge ideas into production systems running at massive scale. We welcome early-career engineers with exceptional technical depth, if you have the intellectual horsepower and hunger to solve problems that don't have textbook answers yet, we want to talk to you.

Description

You will join a team chartered to make Apple's hyper-scale cloud network dramatically more fault-tolerant by building systems that detect and remediate failures automatically, with minimal or no human intervention. This means designing and building the sensing, decision-making, and remediation layers that allow the network to identify anomalies, localize root cause, and take corrective action in real time. You'll work on problems like: how do you detect a degrading network path before it causes an outage? How do you safely automate remediation actions on live production infrastructure without introducing new risk? How do you build a system that learns from past incidents to prevent recurrence? These are open, hard problems, and you will be expected to bring rigorous thinking, creativity, and strong engineering execution to solve them. This role is ideal for someone who loves taking a deep, formal understanding of distributed systems, control theory, graph algorithms, or fault-tolerant design patterns and turning it into resilient, real-world software running in production at enormous scale. You should be comfortable reading research papers and translating relevant ideas into working systems, while also being pragmatic about the constraints of operating in a live, hyper‑scale environment. If you have exceptional technical depth from your academic background, competitive programming, research experience, or personal projects, and you're hungry to work on problems most companies aren't yet equipped to solve, this team is built for you.

Minimum Qualifications

• Bachelor’s degree in Computer Science, Computer Engineering, Electrical Engineering, or equivalent practical experience4 to 6+ years of professional software engineering experience designing, building, and operating production‑grade distributed systems and backend infrastructure
• Deep foundation in computer science fundamentals, including distributed systems architecture, concurrency models, graph algorithms, and systems design
• Strong proficiency in at least one systems‑level or high‑performance language, such as Go, C++, Rust, or Python
• Direct experience designing and building fault‑tolerant mechanisms, including automated self‑healing, active remediation, circuit breaking, load shedding, and blast‑radius mitigation for network services
• Demonstrated ability to model complex failure domains, handle network partitions and split‑brain scenarios, and reason rigorously about system behavior under extreme load and degradation
• Practical experience with chaos engineering, fault injection, simulation‑based testing, and stress testing in production or staging environments
• Track record of technical ownership, including authoring design documents, driving code reviews, and leading post‑mortem root cause analyses

Preferred Qualifications

• Master’s or Ph.D. in Computer Science, Distributed Systems, Networking, or a related technical discipline
• Deep domain knowledge in cloud networking architectures, Software‑Defined Networking (SDN) control planes, L3/L4 routing protocols, overlay networks, and Linux networking constructs (such as eBPF, XDP, or OVS)
• Hands‑on experience implementing or tuning distributed consensus algorithms (such as Raft or Paxos) and closed‑loop control systems or reconciliation controllers
• Experience architecting high‑throughput telemetry pipelines and automated anomaly detection systems for real‑time network health analysis
• Proven ability to digest academic papers and industry research, translating state‑of‑the‑art resilience concepts into production systems
• History of notable open‑source contributions, technical publications, or patent filings in distributed networking and systems reliability