Senior Data Platform Infra Engineer for AI Pipelines

General Intuition; New York (NY); On-site; USD 180,000 - 240,000; Full time

Job summary

General Intuition is building world models and infrastructure that powers massive gameplay data pipelines and training systems. You will own orchestration of GPU clusters, from scheduling to ensuring efficient utilization, while shaping preprocessing pipelines that turn vast video corpora into training-ready data.

You will optimize disk I/O, network throughput, and production inference latency, owning the cost and latency metrics across cloud providers and multi-region deployments.

Qualifications

• Experience with large-scale infrastructure and orchestration across clusters.
• Ability to build and optimize preprocessing pipelines for huge datasets.
• Expertise in managing disk and network I/O as critical constraints.
• Proven track record of optimizing model inference latency in production.
• Comfort with multi-region cloud deployments and infrastructure as code.

Responsibilities

• Own orchestration and GPU clusters, scheduling, utilization, and capacity.
• Build preprocessing pipelines for large corpora of raw gameplay video to training-ready data.
• Treat disk and network I/O as first-class constraints and identify bottlenecks.
• Optimize production inference, including batching and serving runtimes, to meet latency and cost targets.
• Own infrastructure across cloud providers, multi-region deployment, and reliability of systems.

Skills

Orchestration of GPU clusters

Preprocessing pipelines

Disk I/O optimization

Inference latency optimization

Cloud providers & IaC

Python/Go/C++/Rust

Tools

Kubernetes

Terraform

Go

Python

Rust/C++

Job description

General Intuition is building world models and infrastructure that powers massive gameplay data pipelines and training systems. You will own orchestration of GPU clusters, from scheduling to ensuring efficient utilization, while shaping preprocessing pipelines that turn vast video corpora into training-ready data.

You will optimize disk I/O, network throughput, and production inference latency, owning the cost and latency metrics across cloud providers and multi-region deployments.