Senior Data Engineer, Lakehouse & Data Governance

Shield AI; United States; On-site; USD 145,000 - 220,000; Full time

Benefits offered by this job

Bonus

Equity

Benefits

Job summary

Shield AI is seeking a Senior Staff Data Engineer to build and operate an enterprise lakehouse on Databricks. You will own Bronze ingestion and lead Silver quality pipelines, ensuring governed data for multiple domains in a regulated environment.

You will implement secure catalogs, schemas, and job orchestration, while shaping reusable patterns and governance across the platform. This role emphasizes data meaning, quality, and compliance in a fast-moving data platform.

Qualifications

• 12+ years of data engineering experience with production data pipelines.
• Databricks experience including Delta Lake and Spark.
• Unity Catalog knowledge with catalogs, schemas, and access control.
• Experience with batch, CDC, and streaming ingestion.
• CI/CD and deployment automation for data pipelines.
• Strong SQL and data modeling fundamentals.
• Ability to understand business and regulatory context of data.
• Data governance and compliance experience in regulated environments.
• Experience designing pipelines with data domain awareness.
• Comfort with fast-moving platform builds and setting standards.

Responsibilities

• Design ingestion pipelines, batch and streaming, into the Databricks lakehouse using Delta Lake.
• Own Bronze-layer ingestion, including raw landing patterns and load traceability.
• Build Silver-layer pipelines for cleansing, standardization, deduplication, and quality enforcement.
• Define reusable ingestion patterns and templates for other domains.
• Implement secure delivery constructs: catalogs, schemas, and service principals.
• Build and maintain CI/CD pipelines for data assets and workflows.
• Apply data classification and handling requirements within pipelines.
• Create data quality checks that reflect business meaning and domain behavior.
• Quarantine problematic records per defined quality and compliance rules.
• Maintain documentation for source objects, transformations, and limitations.
• Partner with Analytics Engineer and domain teams for reliability and governance.
• Collaborate with domain teams on ownership, data contracts, and onboarding.

Skills

Strong SQL

Data modeling

Regulatory context

Collaboration

Tools

Databricks

Delta Lake

Unity Catalog

Spark (PySpark/Spark SQL)

CI/CD for data

Job description

Shield AI is seeking a Senior Staff Data Engineer to build and operate an enterprise lakehouse on Databricks. You will own Bronze ingestion and lead Silver quality pipelines, ensuring governed data for multiple domains in a regulated environment.

You will implement secure catalogs, schemas, and job orchestration, while shaping reusable patterns and governance across the platform. This role emphasizes data meaning, quality, and compliance in a fast-moving data platform.