GPU Cluster Engineer, Systems & Platform

Sciforium; San Francisco (CA); On-site; USD 190,000 - 270,000; Full time

Benefits offered by this job

Medical insurance

401k plan

Daily meals/snacks

Flexible time off

Competitive salary and equity

Job summary

Sciforium is seeking a GPU Cluster Engineer to own the software stack of GPU clusters—from kernel tuning to ML framework integration. You will define production-ready nodes, automate bring-up, and maintain fleet health while supporting foundation model training and model serving teams.

You will automate image creation, validation suites, and fleet upgrades, leveraging Kubernetes, Slurm, and container tooling to ensure high performance and reliability across the cluster.

Qualifications

• 5+ years in systems/infrastructure engineering with GPU cluster/ML infra experience.
• Bachelor's or Master's in CS/EE or related field.
• Deep Linux internals: kernel modules, DKMS, NUMA, cgroups, system performance tuning.
• Hands-on experience with NVIDIA CUDA and/or ROCm driver stacks on modern accelerators.
• Production Kubernetes experience with GPU workloads; knowledge of HPC schedulers (Slurm/Run:AI).
• Configuration management with Ansible or SaltStack; Git-based workflows.
• Image provisioning/tooling (Packer, MaaS, Foreman, Terraform).
• Experience with distributed filesystems (Lustre, GPFS) and checkpoint I/O optimization.
• Python and Bash for automation; NCCL/RCCL and PyTorch/JAX runtime familiarity.

Responsibilities

• Own node software definition from OS bring-up to production-ready GPU nodes.
• Build automated acceptance suites and validation for fleet readiness.
• Maintain fleet through kernel/driver/toolkit upgrades with minimal disruption.
• Automate health checks, drift detection, and repair handoffs.
• Manage node/configuration via IaC (Ansible/SaltStack) with CI validation and canaries.
• Develop tooling for cluster operations, health reporting, and workflows.
• Operate GPU-enabled Kubernetes and Slurm for serving and training workloads.
• Maintain CUDA/ROCm stacks, container tooling, and reproducible images.

Skills

GPU cluster

Linux internals

Python scripting

Kubernetes

NVIDIA CUDA

Ansible

System performance

Education

Bachelor's or Master's in CS/EE

Tools

NVIDIA Driver Toolkit

ROCm

Packer

Terraform

Job description

Sciforium is an AI infrastructure company developing next-generation multimodal AI models and a proprietary, high-efficiency serving platform. Backed by multi-million-dollar funding and direct sponsorship from AMD with hands-on support from AMD engineers the team is scaling rapidly to build the full stack powering frontier AI models and real-time applications.

About the Role

We are looking for a GPU Cluster Engineer to own the entire software stack of our GPU clusters — from kernel tuning and GPU drivers up through schedulers, containers, and ML frameworks. While our Hardware Operations team keeps the physical machines healthy and connected, you define what a production-ready node looks like in software: you author the images, playbooks, and pipelines that take a freshly provisioned server to a fully validated GPU node, and you keep the fleet consistent, upgradable, and fast. You will serve two demanding customer groups — our foundation model training teams and our model serving/product teams — ensuring both run on correctly configured, well-managed, high-performance infrastructure.

Key Responsibilities

• OS Bring-Up & Node Lifecycle Engineering

  • Golden Images & Automated Bring-Up: Own the node software definition — versioned OS images, kernel tuning (NUMA, hugepages, IRQ affinity, cgroups), GPU/NIC driver stacks — and the automated pipeline that takes a node from base OS to production-ready.

  • Validation & Burn-In: Build automated acceptance suites (DCGM diagnostics, nccl-tests/RCCL tests, bandwidth and topology checks, HPL) that gate every node before it enters a scheduler pool.

  • Fleet Maintenance: Execute rolling kernel/driver/toolkit upgrades with minimal disruption to running workloads; enforce configuration consistency, detect drift, and maintain the driver CUDA/ROCm framework compatibility matrix across the fleet.

  • Self-Healing Operations: Automate detection of unhealthy nodes (Xid/ECC errors, link flaps, thermal throttling), with cordon/drain/reboot/re-image workflows and clean handoff to Hardware Operations for physical repair or RMA.

• Configuration Management & Automation

  • Infrastructure as Code: Manage all node and cluster configuration through Ansible/SaltStack playbooks in Git, with peer-reviewed changes, CI validation, and canary rollouts before fleet-wide deployment.

  • Provisioning Pipelines: Build and maintain image/provisioning tooling (PXE, MaaS, Packer, or similar) so new or re-imaged nodes are reproducible, not hand-crafted.

  • Operational Tooling: Develop Python/Bash tooling for cluster operations, health reporting, and workflow automation.

• Orchestration & Scheduling (Kubernetes & Slurm)

  • Kubernetes for Serving: Deploy and operate GPU-enabled Kubernetes for inference workloads — NVIDIA GPU Operator, device plugins, node feature discovery, topology-aware scheduling, and MIG/MPS partitioning where appropriate.

  • Training Schedulers: Operate Slurm (or Run:AI) for multi-node training — partitions, QoS, preemption, accounting, and container integration (enroot/pyxis).

  • Container Platform: Maintain base images, registries, and the NVIDIA Container Toolkit / ROCm container stack; keep training and serving images lean, current, and reproducible.

• GPU Driver & ML Stack Engineering

  • Driver & Runtime Lifecycle: Build, deploy, and debug the full accelerator stack — NVIDIA (CUDA toolkit, cuDNN, NCCL, Fabric Manager) and AMD (ROCm, RCCL) — including kernel modules (DKMS), GPUDirect RDMA/Storage, and the RDMA software stack (MOFED/DOCA).

  • Framework Environments: Maintain curated, optimized PyTorch and JAX environments with sane dependency and version management for researchers and production services.

  • Distributed Performance: Tune NCCL/RCCL across NVLink/NVSwitch and InfiniBand/RoCE fabrics, ensure topology-aware job placement, and run continuous communication/throughput benchmarks to catch regressions.

• Advanced Debugging & Observability

  • Escalation Point: Own the hard problems — NCCL hangs and timeouts, CUDA memory leaks, ROCm kernel crashes, straggler nodes, and unexplained throughput drops.

  • Observability: Own software-layer monitoring (DCGM exporter, Prometheus/Grafana, alerting) plus job-level GPU utilization and cluster efficiency reporting.

Qualifications

• Must-Haves:

  • 5+ years in systems/infrastructure engineering with significant GPU cluster, HPC, or large-scale ML infrastructure experience.

  • Bachelor's or Master's degree in Computer Science, Computer Engineering, Electrical Engineering, or a related technical field.

  • Deep Linux internals expertise: kernel modules/DKMS, systemd, cgroups, NUMA, and system performance tuning.

  • Hands-on experience with NVIDIA (CUDA) and/or AMD (ROCm) driver and runtime stacks on modern accelerators (H200/B200, MI325x/MI355x class), including kernel-level debugging.

  • Production Kubernetes experience with GPU workloads, plus working knowledge of HPC schedulers (Slurm/Run:AI) — or the reverse (deep Slurm, working K8s).

  • Strong configuration management experience (Ansible or SaltStack) with Git-based, code-reviewed infrastructure workflows.

  • Provisioning and image tooling experience (Packer, MaaS, Foreman, Terraform, or similar) for automated, reproducible node builds.

  • Client-side experience with distributed filesystems (Lustre, GPFS, Weka) and checkpoint I/O optimization.

  • Container fluency: Docker/containerd and the NVIDIA Container Toolkit or ROCm equivalent.

  • Proficiency in Python and Bash for automation and tooling.

  • Working knowledge of NCCL and RDMA networking (InfiniBand/RoCE, GPUDirect) and of PyTorch/JAX runtime behavior.

• Nice-to-Haves:

  • Experience directly supporting foundation model training teams — multi-node job failure debugging, checkpoint pipeline tuning, and framework-level performance triage — ideally in a startup or research-heavy environment.

  • Experience deploying and tuning inference/serving stacks (vLLM, Triton Inference Server, TensorRT-LLM) for latency and throughput targets.

  • GPU/system profiling tools: Nsight Systems/Compute, rocprof, perf, eBPF.

Benefits include

• Medical, dental, and vision insurance

• 401k plan

• Daily lunch, snacks, and beverages

• Flexible time off

• Competitive salary and equity

Equal opportunity

Sciforium is an equal opportunity employer. All applicants will be considered for employment without attention to race, color, religion, sex, sexual orientation, gender identity, national origin, veteran or disability status.