Forward Deployed Infrastructure Engineer - APAC

Hyperbolic; United States; Remoto; USD 120.000 - 180.000; Tempo pieno

Descrizione del lavoro

Hyperbolic Labs is seeking a Technical Support Engineer to embed with reserved customers running multinode GPU clusters. You will stand up their clusters, hand them over, and stay involved to diagnose deep issues across hardware, fabric, and software.

You'll own environments, coordinate with providers to prove fault locations, and build runbooks for L1 teams. Strong Linux, NCCL, and GPU experience are essential for this customer-facing role.

Competenze

• Cluster stand-up and handoff with documentation for the customer's engineers.
• Deep escalations: multinode and NCCL performance, GPU faults, drivers, fabric issues.
• Provider escalation with evidence to drive resolution and clear customer updates.
• Embedded ownership of named accounts: learn workloads, guide changes.
• Proactive monitoring and alerting using Grafana/Prometheus.
• Automate repeat tasks and create runbooks for L1 teams.
• Deep Linux experience and CLI fluency in complex environments.
• Hands-on multinode GPU experience: NCCL, InfiniBand or RoCE, topology/placement.
• Hardware fault triage on GPU nodes: XID, ECC, dmesg, nvidia-smi.
• Experience provisioning and operating GPU clusters with Kubernetes/Slurm.
• Clear, confident communication with customers, including bad news when needed.
• Judgment on when to dig deeper or pivot.

Mansioni

• Stand up, validate, and hand over new customer clusters with proper docs.
• Handle deep escalations for multinode/NCCL performance and hardware faults.
• Coordinate with providers, bringing evidence to resolve faults.
• Own named customer accounts and learn their workloads to guide changes.
• Monitor clusters proactively using Grafana/Prometheus to catch faults early.
• Automate repetitive tasks and build runbooks for L1 teams.
• Maintain deep Linux expertise and CLI fluency in diverse environments.
• Apply hands-on multinode GPU experience with NCCL, InfiniBand/RoCE and topology.
• Triage hardware faults on GPUs: XID, ECC, dmesg, lspci, nvidia-smi.
• Provision GPU clusters with Kubernetes/Slurm and CI/CD tooling.
• Communicate honestly with customers, delivering bad news when necessary.

Conoscenze

Cluster stand-up

Escalation management

Customer-facing

Proactive monitoring

Linux CLI

GPU expertise

Strumenti

NCCL

InfiniBand

RoCE

Grafana

Prometheus

Kubernetes

Slurm

Terraform

Ansible

nvidia-smi

Descrizione del lavoro

Who We Are

Hyperbolic Labs is on a mission to democratize AI by breaking down the barriers to computing power with our Open-Access AI Cloud. By making better use of idle computing resources across the globe, we offer an innovative GPU marketplace and AI inference service that promise affordability and accessibility for all. As pioneers at the intersection of AI and open-source technology, we believe in an open future where AI innovation is limited only by imagination, not by access to resources. We're looking for forward-thinking individuals who share our passion for making AI universally accessible, secure, and affordable. Join us in building a platform that empowers innovators everywhere to turn their visionary AI projects into reality.

About the Role

Our reserved customers run large multinode GPU clusters, and when those clusters misbehave the problem is rarely simple. You are the engineer embedded with those customers: you stand their cluster up, you hand it over, and you stay with it.

You do not own tickets. You own environments. Technical Support Engineers own the ticket lifecycle and pull you in when an issue needs real depth: multinode collective performance, hardware faults, fabric problems, or a provider who needs to be told what is wrong with their hardware.

One thing we will be straight about, because it shapes the job. We aggregate capacity from suppliers rather than owning most of the hardware ourselves. That means a real part of this role is technical liaison work: proving where a fault actually lives, taking it to the provider with evidence, and coordinating the fix on the customer's behalf. The engineers who enjoy this role are the ones who find that interesting rather than frustrating.

Who You Are

• Cluster stand-up and handoff. Build, validate, and benchmark new customer clusters, then hand them over with documentation the customer's own engineers can work from.




• Deep escalations. Multinode and NCCL performance debugging, GPU and hardware faults (XID and ECC errors, lspci, dmesg), driver and fabric issues, container and scheduler problems.




• Provider escalation and coordination. Maintenance windows, RMAs, hung nodes, and disputed fault attribution. You bring the evidence that makes the provider act, and you keep the customer informed while it happens.




• Embedded ownership of named accounts. You are the engineer your customers know by name. You learn their workload, not just their infrastructure, and you tell them what to change before they hit the wall.




• Proactive monitoring. Own the monitoring and alerting we put in front of customer clusters (Grafana, Prometheus) so we find faults before the customer reports them.




• Tooling and pushing work down. Automate the repeat work and turn your own escalations into runbooks the L1 tier can run. Anything you fix three times should stop reaching you.




• Deep Linux experience and total comfort in the CLI, including in someone else's broken environment.




• Hands-on multinode GPU experience: NCCL, InfiniBand or RoCE, collective performance debugging, topology and placement.




• Hardware fault triage on GPU nodes: XID and ECC errors, dmesg, lspci, nvidia-smi, thermal and power faults.




• Experience provisioning and operating GPU clusters with Kubernetes, Slurm, or both.




• Genuinely comfortable customer-facing, including delivering bad news and saying \"this is ours\" or \"this is the provider's\" with confidence.




• Sound judgment on when to keep digging and when to escape.

Preferred Qualifications

• Experience with parallel filesystems (Weka, Lustre, GPFS) and high-performance storage.




• Grafana, Prometheus, or similar observability stacks in production.




• Prior forward deployed engineer, solutions architect, or technical account manager experience.




• Infrastructure as code (Terraform, Ansible) and CI for cluster provisioning.




• Exposure to model training or inference workloads from the practitioner side.

Hyperbolic is an equal opportunity employer. We celebrate diversity and are committed to creating an inclusive environment for all employees.