Senior Data Engineer, Databricks Lakehouse Lead

Shield; United States; On-site; USD 145,000 - 220,000; Full time

Benefits offered by this job

Equity

Bonus

Benefits

Job summary

Shield AI is seeking a Senior Staff Data Engineer to architect and operate a Databricks-based lakehouse. You will own end-to-end data ingestion, processing, and governance across Bronze and Silver layers, ensuring secure, compliant data handling in a regulated environment.

You will implement scalable pipelines, build reusable patterns, and collaborate with analytics teams to enable trusted Gold-layer modeling. A strong focus on data quality and enterprise controls is essential.

Qualifications

• 12+ years of data engineering experience with production pipelines.
• Strong Databricks experience including Delta Lake, Workflows/Jobs, and Spark with PySpark/SQL.
• Knowledge of Unity Catalog including catalogs, schemas, tables, lineage, and access control.
• Experience with batch, CDC, and/or streaming ingestion patterns.
• CI/CD and deployment automation for data pipelines and platform assets.
• Strong SQL skills and data modeling fundamentals.
• Ability to understand business and regulatory data context.
• Experience applying data classification, security, retention in regulated environments.
• Design pipelines with data-domain awareness and governance in mind.
• Comfort in fast-moving platform builds and shaping standards.

Responsibilities

• Design and build ingestion pipelines from enterprise sources into the Databricks lakehouse.
• Own Bronze-layer ingestion, raw landing patterns, metadata capture, and recoverable ingestion.
• Build Silver-layer pipelines for cleansing, standardization, deduplication, conformance, and quality enforcement.
• Define reusable ingestion and transformation patterns and templates.
• Implement secure delivery with catalogs, schemas, service principals, and deployment patterns.
• Build and maintain CI/CD pipelines for data platform assets and workflows.
• Apply data classification, segregation, and handling requirements in pipelines.
• Create data quality controls reflecting business meaning and domain behavior.
• Quarantine, flag, and route problematic data per defined rules.
• Maintain documentation for sources, logic, quality rules, and limitations.
• Collaborate with Analytics Engineer and domain teams to ensure reliable Silver/Gold data.

Tools

Databricks

Delta Lake

Unity Catalog

PySpark

Spark SQL

CI/CD

Infrastructure as Code

Data modeling

Regulatory context

Job description

Shield AI is seeking a Senior Staff Data Engineer to architect and operate a Databricks-based lakehouse. You will own end-to-end data ingestion, processing, and governance across Bronze and Silver layers, ensuring secure, compliant data handling in a regulated environment.

You will implement scalable pipelines, build reusable patterns, and collaborate with analytics teams to enable trusted Gold-layer modeling. A strong focus on data quality and enterprise controls is essential.