Member of Technical Staff - AI Infrastructure

Veeda Innovation; California (MO); Hybrid; USD 180,000 - 240,000; Full time

Job summary

Veeda AI is hiring a Member of Technical Staff - AI Infrastructure to design, deploy, and operate GPU clusters (Slurm-on-Kubernetes) with focus on health, interconnect, storage, and scheduling. You will build data pipelines, security controls, and observability to support large-scale AI workloads.

You will collaborate with Model/Data teams, forecast resources, run PoCs with vendors, and own infra lifecycle from specs to production, including on-call support and cost optimization.

Qualifications

• Bachelor's degree in Computer Science, Computer Engineering, or equivalent hands-on experience in HPC or infrastructure engineering
• Strong troubleshooting skills below the framework layer: low-level Linux networking, kernel and PCIe/NUMA tuning, hardware diagnostics, and network or distributed storage (NFS, NVMe-oF, Lustre, Ceph)
• Proficient in automation and infrastructure-as-code tools (e.g., Ansible, Terraform, Helm, Python/Bash scripting) and treats cluster configuration as reviewed, version-controlled code
• Experience supporting distributed deep learning workloads (PyTorch/NCCL, Ray, DeepSpeed) closely enough to tell an infrastructure fault from a model bug, and able to prove which with a controlled benchmark
• One of the following: Deep hands-on expertise administering Linux-based HPC clusters running Slurm or Kubernetes (job scheduling, cgroups, fair-share policies, topology configuration, GPU device plugins); Experience working with cloud providers on workload profiling, PoC, resource cost strategies; Experience managing security policies. Hands-on-experience with building security layers from overall strategy to IaC.

Responsibilities

• GPU Cluster Operations: Design, deploy, and operate Slurm-on-Kubernetes GPU clusters and manage compute, interconnect, storage, and scheduling resources
• Access Control and Security: Design, deploy, and operate identity provisioning and access control for source code, cluster, knowledge bases, data storages, and dev tools
• Network and Data: Design and maintain distributed data transmission and caching services to deliver data and checkpoints to clusters efficiently
• Observability & Hardware Health: Build telemetry pipeline (Prometheus, Grafana, DCGM, BMC/Redfish) to catch errors and work with vendors to resolve issues
• Resource Forecasting: Determine resource requirements and cadence, lead PoC rounds with vendors for compute/data resources
• Infra-OPEX: Prepare operational expenses forecast and reports, advise on infra spending decisions
• End-to-End Ownership: Lead projects through full software lifecycle including specs, implementation, CI/CD, on-call support, observability

Skills

Linux troubleshooting

HPC infra experience

Automation & IaC

DL workloads support

Education

Bachelor's degree in CS/CE or equivalent

Tools

Ansible

Terraform

Helm

Python/Bash scripting

Job description

Member of Technical Staff - AI Infrastructure

About Us

Veeda AI is building the next generation of multimodal foundation world models for Physical AI. We're a small, fast-moving team of engineers and researchers from leading AI labs, tackling some of the most challenging problems at the intersection of AI, robotics, and embodied intelligence. If you're excited about pushing the boundaries of what's possible with Physical AI, you'll have the opportunity to make an outsized impact from day one.

Responsibilities

• GPU Cluster Operations: Design, deploy, and operate Slurm-on-Kubernetes GPU clusters. Own the compute (CPU/GPU device health), interconnect (nvlink, Infiniband, RoCE), storage (linux file systems, Lustre), and scheduler (topology aware scheduling, QoS, partitions, fair-share) resources.

• Access Control and Security: Design, deploy, and operate identity provisioning and access control for our source code, cluster, knowledge bases, data storages, and dev tools.

• Network and Data: Design and maintain distributed data transmission and caching services that provide just-in-time delivery of data and checkpoints to the clusters in a timely and cost-effective manner. Collaborate with internal stakeholders and external vendors to find the optimal path between data and compute.

• Observability & Hardware Health: Build the telemetry pipeline (Prometheus, Grafana, DCGM, BMC/Redfish) that catches Xid and ECC errors, thermal throttling, and link flaps, and work with cloud vendors to diagnose and resolve issues in timely manner.

• Resource Forecasting: Work with Model and Data teams to determine resource requirements and delivery cadence for the next month/quarter/year. Lead PoC rounds with vendors for compute and data resources.

• Infra-OPEX: Prepare operational expenses forecast and reports. Provide expert opinion on key infra spending decisions.

• End-to-End Ownership: Lead projects through the complete software lifecycle, including technical specs, implementation, CI/CD, on-call support, and production observability.

Requirements

• Bachelor's degree in Computer Science, Computer Engineering, or equivalent hands-on experience in high-performance computing (HPC) or infrastructure engineering.

• Strong troubleshooting skills below the framework layer: low-level Linux networking, kernel and PCIe/NUMA tuning, hardware diagnostics, and network or distributed storage (NFS, NVMe-oF, Lustre, Ceph).

• Proficient in automation and infrastructure-as-code tools (e.g., Ansible, Terraform, Helm, Python/Bash scripting), and treats cluster configuration as reviewed, version-controlled code.

• Experience supporting distributed deep learning workloads (PyTorch/NCCL, Ray, DeepSpeed) closely enough to tell an infrastructure fault from a model bug, and able to prove which with a controlled benchmark.

• One of the following:

  • Deep hands-on expertise administering Linux-based HPC clusters running Slurm or Kubernetes (job scheduling, cgroups, fair-share policies, topology configuration, GPU device plugins).

  • Experience working with cloud providers on workload profiling, Proof of Concepts (PoC), resource cost strategies.

  • Experience managing security policies. Hands-on-experience with building security layers from overall strategy to IaC. Educating fast moving Dev teams of security practices.

Nice to Have

• Experience managing high-density GPU infrastructure (NVIDIA H100/H200, B200, and GB200 NVL72 systems, DGX/HGX architectures, liquid-cooled racks).

• Built container and image supply chains for HPC workloads (Enroot, Pyxis, Docker, custom Kubernetes operators).

• Experience running hybrid capacity, combining owned hardware with cloud or neocloud burst under a single scheduler.

• Experience running secure multi-tenant research environments with SSO, per-team quota, and interactive access that stays fast.

• Fluency with Prometheus, Grafana, and OpenTelemetry, and instrument workloads before its first outage.

• Contributed to Slurm plugins, Kubernetes operators, or open-source cluster and observability tooling.

• Experience with GitLab, especially GitLab CI, for managing infrastructure-as-code and automation pipelines.

• Experience benchmarking fabric, storage, or scheduler performance and publishing results internally or externally to settle a design or procurement decision.