Senior AI Infra Engineer: GPU Compute on Kubernetes

Harell Data; Palo Alto (CA); On-site; USD 180,000 - 260,000; Full time

Job summary

Harell Data is seeking an experienced engineer to lead the compute platform in Palo Alto, focusing on GPU clusters and scalable ML infrastructure. You will own the end-to-end ML pipeline, from data ingestion to deployment, while guiding architectural choices and collaborating closely with customers to translate needs into reliable systems.

You will pair with the CTO and shape the engineering direction, implementing Kubernetes-based orchestration, autoscaling, and efficient resource management to

Qualifications

• 5+ years building and operating production infrastructure, with a focus on ML workloads
• Hands-on experience with Kubernetes on AWS or GCP, ideally with GPU workloads
• Strong CS fundamentals and system design chops
• Comfortable with ambiguity — you’ve worked somewhere where the playbook didn’t exist yet

Responsibilities

• Build the GPU compute layer — orchestration for GPU workloads on Kubernetes, including resource allocation and cost management
• Build the inference layer — model loading, autoscaling, batching, and serving with low latency
• Own the ML pipeline end-to-end — data ingestion, preprocessing, training, fine-tuning, and recovery
• Work directly with customers to debug fine-tuning jobs and improve observability of model performance and resource health
• Own reliability through incident response, on-call duties, and platform uptime as usage grows
• Shape technical direction by leading build-vs-buy decisions and setting engineering standards

Skills

Kubernetes

ML infrastructure

System design

Ambiguity tolerance

GPU compute

AWS/GCP

Tools

GPU clusters

Kubernetes on AWS/GCP

Job description

Harell Data is seeking an experienced engineer to lead the compute platform in Palo Alto, focusing on GPU clusters and scalable ML infrastructure. You will own the end-to-end ML pipeline, from data ingestion to deployment, while guiding architectural choices and collaborating closely with customers to translate needs into reliable systems.

You will pair with the CTO and shape the engineering direction, implementing Kubernetes-based orchestration, autoscaling, and efficient resource management to