Staff Data Platform Engineer - Data Provenance (Equity)

F7d8749a; San Diego (CA); On-site; USD 150,000 - 230,000; Full time

Benefits offered by this job

Equity

Benefits

Bonus

Job summary

Shield AI in San Diego is seeking a Staff Engineer to build the services behind our AI Factory data workflows. You will design and implement data APIs, dataset versioning, search and metadata systems that scale with growing datasets and evolving schemas.

Collaborate with ML engineers to turn recurring data needs into reusable tools, and deliver reliable data services with strong diagnostics and support for large-scale datasets across diverse model workflows.

Qualifications

• Experience designing production data platforms or data-intensive backends.
• Strong Python or equivalent backend language.
• Experience with data indexing, versioning, metadata, and access control.
• Ability to lead technical design and implement substantial parts of a data system.
• Ability to translate ML data needs into reusable data services.

Responsibilities

• Build dataset discovery, catalog and retrieval services, with LanceDB integration.
• Create APIs to pin dataset versions and control access.
• Track data provenance and changes to reproduce results.
• Design schema validation and migration for historical datasets.
• Define identifiers and metadata relationships linking datasets to experiments.
• Collaborate with ML engineers to turn needs into reusable tools.
• Deliver reliable data services with upgrades and diagnostics.

Skills

Data platform design

Python backend

API design

Data indexing and versioning

Schema evolution

Leadership & collaboration

Tools

LanceDB

Arrow/Parquet

Kubernetes

SQL / metadata storage

Job description

Shield AI in San Diego is seeking a Staff Engineer to build the services behind our AI Factory data workflows. You will design and implement data APIs, dataset versioning, search and metadata systems that scale with growing datasets and evolving schemas.

Collaborate with ML engineers to turn recurring data needs into reusable tools, and deliver reliable data services with strong diagnostics and support for large-scale datasets across diverse model workflows.