Lead ML Data Platform Engineer

Build AI

San Francisco (CA)

On-site

USD 180,000 - 240,000

Full time

14 days+

A complete application in a minute

Tailored resume and cover letter, ready to send.

Get past ATS filters

Benefits offered by this job

Competitive pay
Medical, dental, and vision packages
Housing subsidy for SF office nearby
Relocation support to SF or Shenzhen
Wellness benefits
Daily lunch and dinner in our office
Unlimited compute budget
Unlimited Codex and Claude credits
Travel

Job summary

Build AI in San Francisco seeks a lead for the data platform, overseeing on-device capture, upload under flaky bandwidth, and training-ready dataset pipelines. You will design storage, formats, and tooling to support petabyte-scale media for researchers and clients, including coordination with Shenzhen firmware teams.

You will optimize codecs, compression, and storage costs to keep 1080p30 hours affordable while scaling across sites and countries, balancing performance with ROI.

Qualifications

  • Production media path at petabyte scale with video codecs and compression.
  • Experience with high-throughput data pipelines and ETL for large media.
  • Strong software engineering: Python and at least one systems language; Linux.

Responsibilities

  • Own the data platform end-to-end: on-device capture, upload under flaky bandwidth, object storage, training-ready shards.
  • Optimize compression, codecs, and storage for cost-effective 1080p30 hours.
  • Ensure uploads survive poor networks with buffering, batching, retries.
  • Define and manage dataset packaging, versioning, and delivery to researchers.
  • Collaborate with firmware teams to standardize ingest contracts across devices.
  • Model health, cost, and drop rate as we expand to new sites and countries.

Skills

Petabyte-scale storage
Video codecs
Python
Systems programming
Linux
Cost-throughput mindset

Tools

AWS
GCP
Kubernetes
Airflow
Temporal
IaC

Job description

Build AI in San Francisco seeks a lead for the data platform, overseeing on-device capture, upload under flaky bandwidth, and training-ready dataset pipelines. You will design storage, formats, and tooling to support petabyte-scale media for researchers and clients, including coordination with Shenzhen firmware teams.

You will optimize codecs, compression, and storage costs to keep 1080p30 hours affordable while scaling across sites and countries, balancing performance with ROI.

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Staff ML Engineer: Production-Scale AI & Equity
Staff ML Engineer: Production-Scale AI & Equity

Hive • San Francisco (CA)

On-site
USD 200,000 - 300,000
ML Platform Engineer: Build & Scale AI Pipelines
ML Platform Engineer: Build & Scale AI Pipelines

ByteDance • San Jose (CA)

On-site
USD 162,000 - 388,000
Medical insurance
Dental insurance
Vision insurance
+8
Data Platform Engineering Manager — Lead ML Datasets & Big Data
Data Platform Engineering Manager — Lead ML Datasets & Big Data

motional • United States

Hybrid
USD 172,000 - 229,000
Medical, dental, and vision coverage
401k with company match
Pet insurance
+3
Cloud-Scale Backend Engineer for ML Inference
Cloud-Scale Backend Engineer for ML Inference

Praxis, Inc. • San Francisco (CA)

On-site
USD 170,000 - 250,000
Senior ML Engineer, Production-Scale & Impact
Senior ML Engineer, Production-Scale & Impact

Hive • San Francisco (CA)

On-site
USD 160,000 - 250,000
Senior ML Engineer - Remote/Hybrid, Scalable AI Platform
Senior ML Engineer - Remote/Hybrid, Scalable AI Platform

hum.ai • San Francisco (CA)

Remote
USD 180,000 - 280,000
Equity stake
Health insurance
Dental and vision
Lead ML Platform Architect — Scalable AI Infra (Remote)
Lead ML Platform Architect — Scalable AI Infra (Remote)

Motion Recruitment • Massachusetts

Remote
USD 180,000 - 250,000
100% employer-paid benefits
Unlimited PTO
401K
+3
Senior ML Engineer, Data Infrastructure & Pipelines
Senior ML Engineer, Data Infrastructure & Pipelines

Unity Enterprise • Mountain View (CA)

Hybrid
USD 200,000 - 261,000
Comprehensive health insurance
Life and disability insurance
Employee stock ownership
+3
Engineering Manager, Research Data Platform & Datasets
Engineering Manager, Research Data Platform & Datasets

Anthropic Limited • San Francisco (CA)

On-site
USD 405,000 - 850,000
Equity donation matching
Generous vacation and parental leave
Flexible working hours
ML Infrastructure Engineer for Scalable Medical Imaging AI
ML Infrastructure Engineer for Scalable Medical Imaging AI

Epsilon Health • San Francisco (CA)

On-site
USD 150,000 - 250,000