Senior Data Engineer, Databricks Lakehouse & Governance

Shield; United States; Sur place; USD 180 000 - 240 000; Plein temps

Avantages offerts par ce poste

Bonus

Benefits

Equity

Résumé du poste

Shield AI is seeking a Senior Staff Data Engineer to build and operate the enterprise Databricks lakehouse, delivering a governed data foundation for multiple business domains and downstream analytics.

This hands-on role focuses on scalable ingestion, reliable data processing, and robust controls across Bronze and Silver data layers within a highly regulated environment. You will shape patterns and standards adopted by other domains, ensuring secure, compliant data delivery.

Qualifications

• 12+ years of data engineering experience, including hands-on ownership of production data pipelines.
• Strong Databricks experience, including Delta Lake, Databricks Workflows or Jobs, and Spark with PySpark and/or Spark SQL.
• Working knowledge of Unity Catalog, including catalogs, schemas, tables, lineage, and access control concepts.
• Experience with batch, CDC, and/or streaming ingestion patterns and the operational trade-offs associated with each.
• Experience with CI/CD and deployment automation for data pipelines and platform assets, including version control, testing, and controlled promotion across environments.
• Strong SQL skills and solid grounding in data modeling fundamentals.
• Demonstrated ability to understand the business and regulatory context of the data being processed, not just the mechanics of pipeline development.
• Experience applying data classification, segregation, security, retention, or compliance requirements in data engineering workflows within a regulated or security-sensitive environment.

Responsabilités

• Design and build ingestion pipelines, batch and streaming where appropriate, from enterprise source systems into the Databricks lakehouse using Delta Lake.
• Own Bronze-layer ingestion, including raw landing patterns, metadata capture, load traceability, and recoverable ingestion design.
• Build Silver-layer pipelines for cleansing, standardization, deduplication, conformance, and quality enforcement without embedding unauthorized or undocumented KPI logic.
• Define and evolve reusable ingestion and transformation patterns, templates, and engineering standards that other domains can adopt as they onboard to the platform.
• Implement and maintain Databricks platform constructs needed for secure delivery, including catalogs, schemas, service principals, job orchestration, and environment-aware deployment patterns.
• Build and maintain CI/CD pipelines for data platform assets, including ingestion code, transformation logic, workflow definitions, tests, and environment promotion across dev, test, and prod.
• Apply data classification, segregation, and handling requirements within the pipeline design, ensuring sensitive and regulated data is processed in accordance with enterprise controls and access policies.
• Build data quality controls that do more than detect technical failures, including checks that reflect actual business meaning, record integrity, completeness, and expected domain behavior.
• Quarantine, flag, and route problematic records or datasets according to defined quality and compliance rules rather than silently dropping or obscuring issues.
• Maintain documentation for source objects, ingestion logic, applied transformations, data quality rules, and known limitations so downstream teams can trust and use the data correctly.
• Partner with the Analytics Engineer and domain teams to ensure Silver-layer data is reliable, well-governed, and suitable for trusted Gold-layer modeling.
• Collaborate with domain engineering teams to align on ownership boundaries, onboarding patterns, data contracts, and support expectations as new domains are enabled onto the platform.

Connaissances

Databricks

PySpark

Spark SQL

SQL

Data modeling

CI/CD

Data governance

Regulated data handling

Outils

Delta Lake

Unity Catalog

Databricks Workflows

Git

Terraform

Description du poste

Shield AI is seeking a Senior Staff Data Engineer to build and operate the enterprise Databricks lakehouse, delivering a governed data foundation for multiple business domains and downstream analytics.

This hands-on role focuses on scalable ingestion, reliable data processing, and robust controls across Bronze and Silver data layers within a highly regulated environment. You will shape patterns and standards adopted by other domains, ensuring secure, compliant data delivery.