Staff Data Platform Engineer — Reproducible ML Data

Shield

San Diego (CA)

On-site

USD 180,000 - 240,000

Full time

14 days+

A complete application in a minute

Tailored resume and cover letter, ready to send.

Get past ATS filters

Benefits offered by this job

Equity
Bonus
Benefits

Job summary

Shield AI seeks a Staff Engineer to build data APIs and dataset tooling that power AI training workflows. You will design and implement data services that scale with growing datasets and evolving schemas, merging hands-on software development with technical leadership.

You will partner with ML engineers to translate reproducibility needs into reusable data platforms, ensuring secure access, provenance, and robust deployment in diverse environments.

Qualifications

  • Experience designing and implementing production data platforms or data-intensive backend services.
  • Strong software engineering skills in Python or a comparable backend language.
  • Experience with data indexing, versioning, metadata, storage and access control.
  • Experience evolving schemas or APIs while supporting existing users and historical data.
  • Ability to lead technical design and implement substantial parts of the system.
  • Ability to work with data or machine-learning users and translate their needs into reliable services.

Responsibilities

  • Build dataset discovery, catalog and retrieval services, including LanceDB integration for indexing and search.
  • Create APIs that let developers pin an exact dataset version and retrieve only the data they are authorized to access.
  • Track where data came from and how it changed so engineers can understand and reproduce training results.
  • Design schema validation, compatibility and migration mechanisms that preserve access to historical datasets.
  • Define common identifiers and metadata relationships that connect datasets to experiments and model artifacts.
  • Work with machine-learning engineers to understand how they create and use datasets, then turn recurring needs into reusable tools.
  • Deliver reliable data services with tested upgrades, clear diagnostics and support for large datasets.

Skills

Python
Backend development
Data indexing
Versioning
APIs
Leadership

Tools

LanceDB
Arrow
Parquet
Kubernetes

Job description

Shield AI seeks a Staff Engineer to build data APIs and dataset tooling that power AI training workflows. You will design and implement data services that scale with growing datasets and evolving schemas, merging hands-on software development with technical leadership.

You will partner with ML engineers to translate reproducibility needs into reusable data platforms, ensuring secure access, provenance, and robust deployment in diverse environments.

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Staff Data Platform Engineer - Data Provenance (Equity)
Staff Data Platform Engineer - Data Provenance (Equity)

F7d8749a • San Diego (CA)

On-site
USD 150,000 - 230,000
Equity
Benefits
Bonus
Staff Data Platform Engineer: Graph API & DataOps
Staff Data Platform Engineer: Graph API & DataOps

f7d8749a-72d6-407c-bf98-292ea630c73a • San Diego (CA)

On-site
USD 150,000 - 230,000
Staff Engineer, Data Platform (R5659)
Staff Engineer, Data Platform (R5659)

Shield • San Diego (CA)

On-site
USD 180,000 - 240,000
Equity
Bonus
Benefits
Staff Engineer, Data Platform (R5659)
Staff Engineer, Data Platform (R5659)

F7d8749a • San Diego (CA)

On-site
USD 150,000 - 230,000
Equity
Benefits
Bonus
Senior Staff ML Platform Engineer — AI Factory
Senior Staff ML Platform Engineer — AI Factory

F7d8749a • San Mateo (CA)

On-site
USD 210,000 - 260,000
Staff Data Platform Engineer - Equity & Remote Fridays
Staff Data Platform Engineer - Equity & Remote Fridays

Reality Defender Inc. • New York (NY)

On-site
USD 210,000 - 260,000
Healthcare plans with 100% premium for
Dental and Vision plans with 100%
Short/Long-term disability and life保险
+11
Staff Data Platform Engineer — AI Research & Synthetic Data
Staff Data Platform Engineer — AI Research & Synthetic Data

realitydefender • New York (NY)

Remote
USD 150,000 - 230,000
Senior Staff ML Platform Engineer (Kubernetes AI Factory)
Senior Staff ML Platform Engineer (Kubernetes AI Factory)

F7d8749a • San Diego (CA)

On-site
USD 180,000 - 260,000
Senior Data Engineer, Databricks Data Platform
Senior Data Engineer, Databricks Data Platform

F7d8749a • United States

On-site
USD 180,000 - 240,000
Equity
Bonus
Benefits
Senior Staff ML Ops Engineer — AI Platform (Kubernetes)
Senior Staff ML Ops Engineer — AI Platform (Kubernetes)

Precision Labs • San Mateo (CA)

Hybrid
USD 210,000 - 320,000