Senior Data Engineer, Databricks Data Platform

F7d8749a; United States; On-site; USD 180,000 - 240,000; Full time

Benefits offered by this job

Equity

Bonus

Benefits

Job summary

Shield AI is hiring a seasoned Data Engineer to design and operate end-to-end data pipelines in a Databricks lakehouse. You will own Bronze-layer ingestion, build Silver-layer transformations, and define reusable patterns with strong governance across domains.

You will implement platform constructs for secure delivery, build CI/CD for data assets, and ensure data quality, lineage, and compliance in a fast-moving, regulated environment.

Qualifications

• 12+ years of data engineering experience with production data pipelines.
• Deep Databricks experience including Delta Lake, Databricks Workflows, Spark with PySpark/Spark SQL.
• Working knowledge of Unity Catalog catalogs, schemas, tables, lineage, and access control concepts.
• Experience with batch, CDC, and streaming ingestion patterns and the operational trade-offs.
• Experience with CI/CD and deployment automation for data pipelines and platform assets.
• Strong SQL skills and solid grounding in data modeling fundamentals.
• Ability to understand the business and regulatory context of the data being processed.
• Experience applying data classification, segregation, security, retention, or compliance requirements in data engineering workflows.

Responsibilities

• Design and build ingestion pipelines from enterprise source systems into the Databricks lakehouse.
• Own Bronze-layer ingestion including raw landing patterns, metadata capture, load traceability, and recoverable ingestion design.
• Build Silver-layer pipelines for cleansing, standardization, deduplication, conformance, and quality enforcement.
• Define and evolve reusable ingestion and transformation patterns, templates, and engineering standards for onboarding domains.
• Implement and maintain Databricks platform constructs for secure delivery, including catalogs, schemas, service principals, and job orchestration.
• Build and maintain CI/CD pipelines for data platform assets, including ingestion code and tests across environments.
• Apply data classification, segregation, and handling requirements within pipeline design.
• Build data quality controls that reflect business meaning, record integrity, completeness, and domain behavior.
• Quarantine, flag, and route problematic records according to quality and compliance rules.
• Maintain documentation for source objects, ingestion logic, transformations, data quality rules, and limitations.
• Partner with Analytics Engineer and domain teams to ensure Silver/gold data reliability and governance.
• Collaborate with domain teams to align ownership boundaries, onboarding patterns, and data contracts.

Skills

Databricks

PySpark

Spark SQL

Unity Catalog

SQL

Data Modeling

Streaming

Batch processing

CDC

CI/CD

Deployment automation

Regulated environment

12+ years data engineering

Databricks Workflows

Education

Bachelor's or Master’s in a technical field

Tools

Databricks Workflows

Job description

Shield AI is hiring a seasoned Data Engineer to design and operate end-to-end data pipelines in a Databricks lakehouse. You will own Bronze-layer ingestion, build Silver-layer transformations, and define reusable patterns with strong governance across domains.

You will implement platform constructs for secure delivery, build CI/CD for data assets, and ensure data quality, lineage, and compliance in a fast-moving, regulated environment.