Founding Senior Infra Engineer, Distributed Infra

Goaly AI

Palo Alto (CA)

Hybride

USD 180 000 - 240 000

Plein temps

14 jours+

Obtenez une réponse de cet employeur

Votre CV et votre lettre de motivation, adaptés exactement à ce que l’employeur recherche.

Passez les filtres ATS

Avantages offerts par ce poste

Meals and office benefits

Résumé du poste

Goaly AI in Palo Alto is building stealth infrastructure to enable AI research at scale. You will own accelerator clusters across their lifecycles—from bringing capacity online to upgrades, failures, recovery, and retirement.

This hands-on role spans cloud and data-center environments, cluster control planes, networking, storage, security, observability, and automation. You will partner with hardware and cloud providers and with Training, RL Systems, Post-Training, Inference, and Security teams

Qualifications

  • Deep expertise in distributed systems, reliability, and cloud platforms.
  • Strong programming ability in Python, Go, Rust.
  • Hands-on experience with Linux, containers, Kubernetes, and IaC.
  • Experience with cloud providers (AWS/GCP/Azure).

Responsabilités

  • Design, build, and operate control-plane services and IaC for provisioning, upgrades, and recovery.
  • Coordinate capacity online with cloud providers, datacenter, and hardware partners.
  • Improve cluster scalability, health signals, and fault tolerance.
  • Lead incident response and blameless postmortems for cluster failures.
  • Collaborate with Training, RL Systems, Post-Training, and Inference teams to plan compute roadmap.

Connaissances

Distributed systems
Cloud platforms
Python
Go
Rust
Linux
IaC
Kubernetes

Outils

Terraform
Kubernetes

Description du poste

About Us

We’re building toward a world where every company can become its own AI lab. Goaly is a stealth AI startup founded by ex-Meta Superintelligence Labs engineers and researchers. Our mission is to dramatically lower the cost, time, and talent barriers to building proprietary AI — and make each generation of models faster and cheaper to build than the last.

About The Role

You will own systems across the lifecycle of our accelerator clusters—from bringing capacity online and upgrading fleets to detecting failures, recovering safely, and retiring capacity. Your work will determine how quickly researchers can start experiments, how efficiently expensive hardware is used, and how reliably long-running workloads complete.

This is a hands-on infrastructure role spanning cloud and datacenter environments, cluster control planes, networking, storage, security, observability, and automation. You will partner with hardware and cloud providers and our Training, RL Systems, Post-Training, Inference, and Security teams to turn heterogeneous compute into a dependable platform. Depending on experience, you may lead multi-quarter initiatives and help set technical direction.

What You’ll Do
  • Design, build, and operate control-plane services and infrastructure-as-code for provisioning, configuration, validation, upgrades, expansion, draining, recovery, and decommissioning; make every change repeatable, auditable, and safe to roll back.
  • Bring new accelerator capacity online on schedule by coordinating dependencies across cloud providers, datacenter and hardware partners, networking, storage, security, and internal compute consumers.
  • Build high-bandwidth, topology-aware connectivity within and across clusters; diagnose performance and reliability issues spanning hosts, switches, routing, transport, collective communication, and workload placement.
  • Make clusters secure by default through identity and access controls, network policy, workload isolation, host and container hardening, secrets management, and trusted software and image supply chains.
  • Improve fleet scalability, consistency, and fault tolerance by defining health signals, automating remediation, reducing configuration drift, and designing for partial failure.
  • Establish service-level objectives and observability for cluster readiness, provisioning time, usable capacity, job-start latency, infrastructure-caused failures, utilization, and recovery time.
  • Lead incident response and blameless postmortems for cluster failures; turn recurring operational pain into automation, safer defaults, and simpler system boundaries.
  • Work directly with Training, RL Systems, Post-Training, and Inference engineers to debug cross-layer failures and shape a long-term compute, data, networking, and capacity roadmap.
You may be a good fit if you have
  • Deep expertise in distributed systems, reliability, and cloud platforms (e.g., Kubernetes, IaC, AWS/GCP/Azure).
  • Strong programming ability in Python, Go, Rust, or another language suited to reliable infrastructure services and automation.
  • Hands-on experience with Linux, containers, Kubernetes or another cluster scheduler, infrastructure-as-code, and at least one major cloud platform or substantial bare-metal environment.
  • A practical understanding of networking, storage, identity, observability, and reliability, with the ability to trace a failure across multiple layers of a complex system.
  • Experience designing systems for safe rollout, fault isolation, idempotency, capacity growth, and recovery from partial or large-scale failures.
  • High ownership and clear communication, including comfort coordinating multi-team projects and participating in a healthy on-call rotation.
Strong pluses
  • Experience operating large GPU or accelerator fleets for distributed model training, inference, scientific computing, or another communication-intensive workload.
  • Depth in Kubernetes internals, custom controllers or operators, device plugins, cluster autoscaling, scheduler extensions, Slurm, or comparable orchestration systems.
  • Experience with high-performance networking such as RDMA, InfiniBand, RoCE, BGP, cloud interconnects, multi-NIC hosts, CNI or eBPF networking, or topology-aware placement.
  • Experience with Terraform, workflow orchestration, and automated qualification of hosts, drivers, firmware, networks, and new hardware.
  • Knowledge of GPU systems, NCCL, NVLink or NVSwitch, and the failure modes of large distributed jobs.
How We Work
  • Mission first. We choose work for its impact on the mission and take responsibility for the outcome, not just our assigned tasks.
  • High agency. We identify what is missing, form a plan, and move without waiting for perfect clarity.
  • Speed with rigor. We ship, measure, and iterate quickly while protecting correctness, safety, and reliability.
  • Flexible scope. We cross team and technical boundaries when that is the fastest way to solve the real problem.
  • Low ego, high standards. We give direct feedback, change our minds when the evidence changes, and help the whole team win.
  • Continuous learning. The stack changes quickly; we are willing to learn unfamiliar systems, methods, and domains as the work demands.
Location, visa sponsorship & benefits
  • Location-based hybrid policy. This is a location-based hybrid role. We currently expect all staff to work from one of our offices at least three days per week. Exact office options will be confirmed during the recruiting process.
  • Visa sponsorship. We do sponsor visas. However, we cannot successfully sponsor a visa for every role and every candidate. If we make you an offer, we will make every reasonable effort to secure the necessary visa, and we retain immigration counsel to support the process.
  • Meals and office benefits. We provide complimentary lunch and dinner in our offices, along with snacks and beverages.

A note on qualifications. We care more about exceptional evidence than a perfect keyword match. If the work excites you and you can show unusual strength, learning speed, or ownership, we encourage you to apply even if your background does not match every preferred qualification.

Equal opportunity

We are an equal opportunity employer and consider qualified applicants without regard to any characteristic protected by applicable law. Reasonable accommodations are available throughout the hiring process.

Obtenez votre examen gratuit et confidentiel de votre CV.

ou faites glisser et déposez votre fichier ici.

Similar jobs

Postes similaires à comparer

Founding Senior Infra Engineer, Distributed Infra
Founding Senior Infra Engineer, Distributed Infra

Goaly • Palo Alto (CA)

Hybride
USD 140 000 - 180 000
Meals and office benefits
Founding Senior AI Infrastructure Engineer
Founding Senior AI Infrastructure Engineer

Goaly • Palo Alto (CA)

Sur place
USD 140 000 - 210 000
Software Engineer, Compute Infrastructure
Software Engineer, Compute Infrastructure

OpenAI • Los Angeles (CA)

Sur place
USD 230 000 - 405 000
Equity
Flexible work environment
Health benefits
New Grad MST: AI Systems Engineer
New Grad MST: AI Systems Engineer

Goaly • Menlo Park (CA)

Hybride
USD 120 000 - 170 000
Meals and office benefits
Staff Engineer, Frontier AI Training Platform
Staff Engineer, Frontier AI Training Platform

Prime Intellect, Inc. • États-Unis

À distance
USD 150 000 - 300 000
Cash compensation $150K–$300K with equ
Flexible work arrangement (remote or S
Visa sponsorship and relocation
+3
Founding Infrastructure Engineer
Founding Infrastructure Engineer

Peano AI • Palo Alto (CA)

Sur place
USD 140 000 - 210 000
Senior Backend Engineer, Managed Services
Senior Backend Engineer, Managed Services

Lightning AI • New York (NY)

Hybride
USD 180 000 - 250 000
Health Coverage
Equity
401(k) matching
+8
Member of Technical Staff - Training Platform
Member of Technical Staff - Training Platform

Prime Intellect • États-Unis

À distance
USD 150 000 - 300 000
Cash compensation $150K–$300K with equ
Flexible work arrangement (remote or S
Visa sponsorship and relocation
+3
(Staff / Senior Staff) AI Inference Reliability Engineer, Compute Infra
(Staff / Senior Staff) AI Inference Reliability Engineer, Compute Infra

FDE Times • San Francisco (CA)

Sur place
USD 250 000 - 320 000
Startup equity
Health insurance
Competitive benefits
Founding Infrastructure Engineer
Founding Infrastructure Engineer

Peanoai • Palo Alto (CA)

Sur place
USD 180 000 - 260 000