Member of Technical Staff - AI Infrastructure

Veeda AI; Seattle (WA); On-site; USD 180,000 - 240,000; Full time

Job summary

Veeda AI in Seattle is seeking an infrastructure engineer to design, deploy, and operate state-of-the-art GPU clusters and data services for multimodal AI workloads. You will manage compute health, interconnects, storage, and scheduler resources across Slurm and Kubernetes environments.

The role emphasizes security, observability, and scalability, collaborating with model and data teams to plan capacity and tooling. A strong Linux/HPC background and IaC automation are essential for success.

Qualifications

• Bachelor’s degree in Computer Science, Computer Engineering, or equivalent hands-on experience in HPC infra.

Responsibilities

• Design, deploy, and operate Slurm-on-Kubernetes GPU clusters with hardware health monitoring.
• Implement access control for source code, clusters, data storages, and dev tools.
• Build distributed data transmission and caching services for timely data delivery.
• Develop telemetry pipelines with Prometheus, Grafana, DCGM, and Redfish.
• Forecast resources with Model and Data teams and run PoCs with vendors.
• Prepare infra-OPEx forecasts and provide guidance on spend decisions.
• Lead end-to-end projects from specs to CI/CD and on-call production support.

Skills

Linux networking

Kernel tuning

Automation tooling

Python scripting

HPC clusters

Slurm Kubernetes admin

PyTorch NCCL

Cloud workload profiling

Security IaC

Education

Bachelor's degree

Tools

Ansible

Terraform

Helm

GitLab CI

Docker

Prometheus

Grafana

OpenTelemetry

Slurm

Kubernetes

Ceph

Lustre

NFS

NVMe-oF

Job description

About Us

Veeda AI is building the next generation of multimodal foundation world models for Physical AI. We're a small, fast-moving team of engineers and researchers from leading AI labs, tackling some of the most challenging problems at the intersection of AI, robotics, and embodied intelligence. If you're excited about pushing the boundaries of what's possible with Physical AI, you'll have the opportunity to make an outsized impact from day one.

About Us

Veeda AI is building the next generation of multimodal foundation world models for Physical AI. We're a small, fast-moving team of engineers and researchers from leading AI labs, tackling some of the most challenging problems at the intersection of AI, robotics, and embodied intelligence. If you're excited about pushing the boundaries of what's possible with Physical AI, you'll have the opportunity to make an outsized impact from day one.

Responsibilities

• GPU Cluster Operations: Design, deploy, and operate Slurm-on-Kubernetes GPU clusters. Own the compute (CPU/GPU device health), interconnect (nvlink, Infiniband, RoCE), storage (linux file systems, Lustre), and scheduler (topology aware scheduling, QoS, partitions, fair-share) resources.
• Access Control and Security: Design, deploy, and operate identity provisioning and access control for our source code, cluster, knowledge bases, data storages, and dev tools.
• Network and Data: Design and maintain distributed data transmission and caching services that provide just-in-time delivery of data and checkpoints to the clusters in a timely and cost-effective manner. Collaborate with internal stakeholders and external vendors to find the optimal path between data and compute.
• Observability & Hardware Health: Build the telemetry pipeline (Prometheus, Grafana, DCGM, BMC/Redfish) that catches Xid and ECC errors, thermal throttling, and link flaps, and work with cloud vendors to diagnose and resolve issues in timely manner.
• Resource Forecasting: Work with Model and Data teams to determine resource requirements and delivery cadence for the next month/quarter/year. Lead PoC rounds with vendors for compute and data resources.
• Infra-OPEX: Prepare operational expenses forecast and reports. Provide expert opinion on key infra spending decisions.
• End-to-End Ownership: Lead projects through the complete software lifecycle, including technical specs, implementation, CI/CD, on-call support, and production observability.

Requirements

• Bachelor's degree in Computer Science, Computer Engineering, or equivalent hands-on experience in high-performance computing (HPC) or infrastructure engineering.
• Strong troubleshooting skills below the framework layer: low-level Linux networking, kernel and PCIe/NUMA tuning, hardware diagnostics, and network or distributed storage (NFS, NVMe-oF, Lustre, Ceph).
• Proficient in automation and infrastructure-as-code tools (e.g., Ansible, Terraform, Helm, Python/Bash scripting), and treats cluster configuration as reviewed, version-controlled code.
• Experience supporting distributed deep learning workloads (PyTorch/NCCL, Ray, DeepSpeed) closely enough to tell an infrastructure fault from a model bug, and able to prove which with a controlled benchmark.
• One of the following:
  • Deep hands-on expertise administering Linux-based HPC clusters running Slurm or Kubernetes (job scheduling, cgroups, fair-share policies, topology configuration, GPU device plugins).
  • Experience working with cloud providers on workload profiling, Proof of Concepts (PoC), resource cost strategies.
  • Experience managing security policies. Hands‑on‑experience with building security layers from overall strategy to IaC. Educating fast moving Dev teams of security practices.

Nice to Have

• Experience managing high-density GPU infrastructure (NVIDIA H100/H200, B200, and GB200 NVL72 systems, DGX/HGX architectures, liquid‑cooled racks).
• Built container and image supply chains for HPC workloads (Enroot, Pyxis, Docker, custom Kubernetes operators).
• Experience running hybrid capacity, combining owned hardware with cloud or neocloud burst under a single scheduler.
• Experience running secure multi‑tenant research environments with SSO, per‑team quota, and interactive access that stays fast.
• Fluency with Prometheus, Grafana, and OpenTelemetry, and instrument workloads before its first outage.
• Contributed to Slurm plugins, Kubernetes operators, or open‑source cluster and observability tooling.
• Experience with GitLab, especially GitLab CI, for managing infrastructure‑as‑code and automation pipelines.
• Experience benchmarking fabric, storage, or scheduler performance and publishing results internally or externally to settle a design or procurement decision.