Lead ML Data Platform Engineer

Build AI; San Francisco (CA); On-site; USD 180,000 - 240,000; Full time

Benefits offered by this job

Competitive pay

Medical, dental, and vision packages

Housing subsidy for SF office nearby

Relocation support to SF or Shenzhen

Wellness benefits

Daily lunch and dinner in our office

Unlimited compute budget

Unlimited Codex and Claude credits

Travel

Job summary

Build AI in San Francisco seeks a lead for the data platform, overseeing on-device capture, upload under flaky bandwidth, and training-ready dataset pipelines. You will design storage, formats, and tooling to support petabyte-scale media for researchers and clients, including coordination with Shenzhen firmware teams.

You will optimize codecs, compression, and storage costs to keep 1080p30 hours affordable while scaling across sites and countries, balancing performance with ROI.

Qualifications

• Production media path at petabyte scale with video codecs and compression.
• Experience with high-throughput data pipelines and ETL for large media.
• Strong software engineering: Python and at least one systems language; Linux.

Responsibilities

• Own the data platform end-to-end: on-device capture, upload under flaky bandwidth, object storage, training-ready shards.
• Optimize compression, codecs, and storage for cost-effective 1080p30 hours.
• Ensure uploads survive poor networks with buffering, batching, retries.
• Define and manage dataset packaging, versioning, and delivery to researchers.
• Collaborate with firmware teams to standardize ingest contracts across devices.
• Model health, cost, and drop rate as we expand to new sites and countries.

Skills

Petabyte-scale storage

Video codecs

Python

Systems programming

Linux

Cost-throughput mindset

Tools

AWS

GCP

Kubernetes

Airflow

Temporal

IaC

Job description

Build AI in San Francisco seeks a lead for the data platform, overseeing on-device capture, upload under flaky bandwidth, and training-ready dataset pipelines. You will design storage, formats, and tooling to support petabyte-scale media for researchers and clients, including coordination with Shenzhen firmware teams.

You will optimize codecs, compression, and storage costs to keep 1080p30 hours affordable while scaling across sites and countries, balancing performance with ROI.