Technical Lead & Manager, ML Datasets Platform

ProducePay; United States; Hybrid; USD 200,000 - 280,000; Full time

Benefits offered by this job

Equity awards

Health and Wellness benefits

Unlimited Vacation

Work from Home support

Catered meals and snacks

Team building events

Job summary

Waabi is seeking a hands-on technical leader to head the ML Datasets Platform. You will define APIs, tooling, and storage formats for versioned, updatable datasets, while growing the team and roadmap.

You’ll partner with ML Training Platform and model teams to migrate tooling into the platform. You will design scalable data infrastructure, enable reproducible ETL workflows, and own dataloading tools with observability across multi-cloud training.

Qualifications

• Deep ML dataset experience: designed and built large-scale data infrastructure for ML training (storage formats, dataset versioning, ETL, dataloading).
• Strong software engineering: excellent Python skills, PyTorch, distributed systems, cloud storage, data processing frameworks (Spark, Ray, Beam).
• Proven technical leadership: led a platform/infrastructure used by multiple teams and drove adoption.

Responsibilities

• Lead and grow the ML Dataset Platform team: set direction, roadmap, hire, mentor engineers.
• Define APIs, tools and storage formats for ML datasets across training workloads.
• Build tooling to manage datasets at scale with versioning and incremental updates.
• Design APIs for training-time transformations and reproducible ETL vs on-the-fly transforms.
• Own common dataloading tools and monitor bottlenecks and throughput.
• Streamline data access for multi-cloud training with cost and latency controls.
• Make data lineage a first-class property across datasets.
• Partner with ML Training Platform and development teams to migrate tooling and align roadmaps.
• Drive adoption by coordinating with model teams to migrate training data onto the platform.

Skills

ML dataset infra

Python & PyTorch

Distributed systems

Cloud storage

ETL pipelines

Leadership

People management

Team collaboration

Tools

Spark

Ray

Beam

Job description

Waabi is seeking a hands-on technical leader to head the ML Datasets Platform. You will define APIs, tooling, and storage formats for versioned, updatable datasets, while growing the team and roadmap.

You’ll partner with ML Training Platform and model teams to migrate tooling into the platform. You will design scalable data infrastructure, enable reproducible ETL workflows, and own dataloading tools with observability across multi-cloud training.