Senior Data Engineer, Lakehouse & Governance

Shield

United States

Remote

USD 180,000 - 230,000

Full time

14 days+

A complete application in a minute

Tailored resume and cover letter, ready to send.

Get past ATS filters

Job summary

Shield AI is seeking a Senior Staff Data Engineer to design and operate the enterprise lakehouse on Databricks, creating a governed data foundation for multiple domains. You will own Bronze ingestion, build Silver pipelines, and implement controls across regulated data environments.

The role is hands-on, requiring deep understanding of data meaning, sensitivity, and compliance. Join a fast-moving platform build and help shape standard patterns and governance across teams.

Qualifications

  • 12+ years of data engineering experience, including hands-on ownership of production data pipelines.
  • Strong Databricks experience, including Delta Lake, Databricks Workflows or Jobs, and Spark with PySpark and/or Spark SQL.
  • Working knowledge of Unity Catalog, including catalogs, schemas, tables, lineage, and access control concepts.
  • Experience with batch, CDC, and/or streaming ingestion patterns and the operational trade-offs associated with each.
  • Experience with CI/CD and deployment automation for data pipelines and platform assets, including version control, testing, and controlled promotion across environments.
  • Strong SQL skills and grounding in data modeling fundamentals, even if dimensional modeling is not the primary responsibility.

Responsibilities

  • Design and build ingestion pipelines, batch and streaming where appropriate, into the Databricks lakehouse using Delta Lake.
  • Own Bronze-layer ingestion including raw landing patterns and load traceability.
  • Build Silver-layer pipelines for cleansing, standardization, deduplication, conformance, and quality enforcement.
  • Define and evolve reusable ingestion patterns, templates, and standards for onboarding to the platform.
  • Implement and maintain Databricks platform constructs for secure delivery (catalogs, schemas, service principals, job orchestration).
  • Build and maintain CI/CD pipelines for data platform assets, including ingestion code, transformation logic, workflow definitions, tests, and environment promotion.
  • Apply data classification, segregation, and handling requirements to ensure sensitive data is processed per controls and policies.
  • Build data quality controls that reflect business meaning, completeness, and domain behavior.
  • Quarantine and route problematic records according to quality and compliance rules.
  • Maintain documentation for source objects, ingestion logic, transformations, and data quality rules.
  • Partner with Analytics Engineer and domain teams to ensure Silver-layer data is reliable for Gold-layer modeling.
  • Collaborate with domain teams to align ownership, onboarding patterns, data contracts, and support expectations.

Skills

Databricks
SQL
Data modeling
CI/CD
PySpark
Spark SQL
Unity Catalog
Data governance
Streaming ingestion

Tools

Delta Lake
Databricks Workflows
Spark

Job description

Shield AI is seeking a Senior Staff Data Engineer to design and operate the enterprise lakehouse on Databricks, creating a governed data foundation for multiple domains. You will own Bronze ingestion, build Silver pipelines, and implement controls across regulated data environments.

The role is hands-on, requiring deep understanding of data meaning, sensitivity, and compliance. Join a fast-moving platform build and help shape standard patterns and governance across teams.

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Senior Data Engineer, Databricks Data Platform
Senior Data Engineer, Databricks Data Platform

F7d8749a • United States

On-site
USD 180,000 - 240,000
Equity
Bonus
Benefits
Senior Data Platform Architect for Lakehouse & AI
Senior Data Platform Architect for Lakehouse & AI

Qualified Health PBC • Oregon

Hybrid
USD 215,000 - 235,000
Equity
Health insurance
Hybrid work options
Senior Data Platform Architect - Lakehouse, AI, Governance
Senior Data Platform Architect - Lakehouse, AI, Governance

SanDisk • Milpitas (CA)

On-site
USD 170,000 - 230,000
Senior Data Platform Engineer – Lakehouse, Databricks
Senior Data Platform Engineer – Lakehouse, Databricks

Stateside • Adelphi (MD)

Hybrid
USD 119,000 - 129,000
Generous Time Off
Health Coverage
Retirement Plans
Senior Data Platform Engineer — Lakehouse & AI Pipelines
Senior Data Platform Engineer — Lakehouse & AI Pipelines

Qualified Health PBC • United States

Hybrid
USD 215,000 - 235,000
Equity
Medical/Dental/Vision insurance
Flexible working hours
+1
Senior Databricks Data Engineer — AI, Lakehouse & Governance
Senior Databricks Data Engineer — AI, Lakehouse & Governance

PharmaLink, LLC • Largo (FL)

On-site
USD 120,000 - 180,000
Senior Data Platform Architect — Lakehouse & AI Pipelines
Senior Data Platform Architect — Lakehouse & AI Pipelines

Qualified Health PBC • United States

Hybrid
USD 215,000 - 235,000
Equity
Health insurance
Flexible working hours
+1
Senior Data Platform Architect — AI + Lakehouse Leader
Senior Data Platform Architect — AI + Lakehouse Leader

Qualified Health • United States

Hybrid
USD 215,000 - 235,000
Equity
Medical Insurance
Dental & Vision
+3
Databricks Lakehouse Architect: AI & Data Governance
Databricks Lakehouse Architect: AI & Data Governance

CCS IT LIMITED • Washington

On-site
USD 160,000 - 210,000
Databricks Data Engineer — Lakehouse & Governance Expert
Databricks Data Engineer — Lakehouse & Governance Expert

UNAVAILABLE • McLean (VA)

On-site
USD 120,000 - 170,000