Senior Storage Engineer

Cotidal; Palo Alto (CA); On-site; USD 180,000 - 240,000; Full time

Benefits offered by this job

Medical, dental, and vision coverage

Unlimited PTO

Relocation support

Job summary

Cotidal Limited is hiring a Senior Storage Engineer to design, deploy, operate and troubleshoot AI infrastructure storage. You will focus on performance, availability and data protection for large-scale GPU clusters, with immediate work on VAST Data and parallel file systems.

You will collaborate with AI infrastructure, compute, networking and platform teams to resolve bottlenecks, perform upgrades and implement monitoring and automation for the storage stack.

Qualifications

• 10–15 years of professional storage engineering and infrastructure experience.
• Strong production experience with enterprise and distributed storage platforms, with hands-on expertise in VAST Data, WEKA or comparable storage.
• Familiarity with Dell PowerScale/Isilon, NetApp ONTAP, Pure Storage or IBM Storage Scale (GPFS).
• Experience with GPU cluster workloads, parallel I/O, dataset ingestion, checkpointing, performance tuning and bottleneck analysis.
• Knowledge of NFS, SMB, NVMe/TCP, NVMe/RDMA, S3-compatible object storage, RDMA/RoCEv2, TCP/IP and 100G/200G/400G Ethernet.
• Linux administration and experience with Python, Bash, Ansible or Terraform.
• Experience with telemetry, Prometheus/Grafana, incident response, change management and capacity planning.

Responsibilities

• Design, deploy, operate and maintain large-scale storage for AI/HPC and GPU clusters.
• Support parallel file systems, scale-out NAS and distributed storage architectures.
• Manage and troubleshoot VAST Data, WEKA and other distributed storage platforms.
• Optimize throughput, latency, IOPS, checkpointing and data ingestion for AI workloads.
• Configure and troubleshoot NFS, SMB, NVMe-oF and S3-compatible storage access.
• Manage capacity, replication, snapshots, data protection and disaster recovery.

Skills

Storage engineering

VAST Data

WEKA

NFS

SMB

NVMe/OF

S3-compatible

Python

Bash

Ansible

Terraform

Prometheus

Grafana

Capacity planning

Tools

VAST Data

WEKA

Dell PowerScale/Isilon

NetApp ONTAP

Pure Storage

IBM Storage Scale (GPFS)

Lustre

BeeGFS

GPFS

NVIDIA DGX

Job description

WHO WE ARE

Supercomputing built for AI.

Cotidal is building the computing infrastructure that AI companies and research labs need to train and run their models. Starting with TPUs, we design the hardware and software together around their workloads, with the goal of making powerful AI chips easier to use at scale.

Our mission is endless compute for humanity. Over time, we want customers to choose across chip stacks without rebuilding their infrastructure.

THE ROLE

We’re hiring a Senior Storage Engineer to design, deploy, operate and troubleshoot the storage that supports Cotidal’s AI infrastructure. You will focus on performance, availability, scalability and data protection, with immediate work that includes day-to-day VAST Data operations.

You will bring hands‑on experience with VAST Data, WEKA or comparable high‑performance storage and work with AI infrastructure, compute, networking and platform engineering teams. Your work will connect distributed storage, GPU workloads and high-speed networks, from deployment and capacity planning to diagnosing production bottlenecks.

WHAT YOU WILL DO

• Design, deploy, operate and maintain large‑scale storage for AI/HPC and GPU clusters. Support parallel file systems, scale‑out NAS and distributed storage architectures.

• Manage and troubleshoot VAST Data, WEKA and other distributed storage platforms. Optimize throughput, latency, IOPS, checkpointing and data ingestion for AI workloads.

• Configure and troubleshoot NFS, SMB, NVMe‑oF and S3‑compatible storage access. Support storage networking over high‑speed Ethernet and RDMA/RoCEv2.

• Manage capacity, replication, snapshots, data protection and disaster recovery so storage can meet performance and availability needs as deployments grow.

• Investigate bottlenecks across compute, networking and storage, lead incident root‑cause analysis, and work with AI infrastructure, compute, networking and platform teams to resolve issues.

• Perform upgrades, firmware maintenance and capacity expansions. Build and maintain the monitoring and automation needed to operate the storage infrastructure.

WHAT WE ARE LOOKING FOR

• 10–15 years of professional storage engineering and infrastructure experience, including designing, deploying, operating and troubleshooting enterprise and high‑performance storage.

• Strong production experience with enterprise and distributed storage platforms, and hands‑on expertise with VAST Data, WEKA or comparable high‑performance storage.

• Familiarity with Dell PowerScale/Isilon, NetApp ONTAP, Pure Storage or IBM Storage Scale (GPFS).

• Experience with GPU cluster workloads, parallel I/O, dataset ingestion, checkpointing, performance tuning and bottleneck analysis.

• Knowledge of NFS, SMB, NVMe/TCP, NVMe/RDMA, S3‑compatible object storage, RDMA/RoCEv2, TCP/IP and 100G/200G/400G Ethernet.

• Linux administration and experience with Python, Bash, Ansible or Terraform.

• Experience with telemetry, Prometheus/Grafana, incident response, change management and capacity planning.

• The ability to troubleshoot critical performance and availability issues independently, with strong communication, documentation and cross‑functional collaboration skills.

ESPECIALLY VALUABLE

Experience in one or more of the following:

• Deep VAST Data and WEKA architecture, deployment and performance tuning.

• Lustre, BeeGFS, IBM Storage Scale (GPFS) or Ceph.

• Integration with NVIDIA DGX/HGX or AMD GPU clusters, and NVIDIA GPUDirect Storage (GDS).

• Kubernetes storage and CSI drivers, multi‑site replication and data mobility.

• Hyperscale or cloud AI infrastructure and petabyte‑to exabyte‑scale datasets.

WHY JOIN COTIDAL

Help build and run the storage foundation for AI infrastructure. Your work will connect data ingestion, checkpointing and storage performance with the reliability of the systems our customers use, and turn production experience into better designs and operating practices.

Help shape the foundations of an early company through direct discussion and work across AI infrastructure, compute, networking and platform engineering.

LOCATION & WHAT WE OFFER

Location: Palo Alto, in person.

Compensation: Competitive salary and equity.

Visa sponsorship: We sponsor work visas and help you navigate the process.

Benefits: Medical, dental, and vision coverage, unlimited PTO, and relocation support as needed.