Senior Staff Data Engineer, Databricks (R6508)

F7d8749a; United States; Vor Ort; USD 180.000 - 240.000; Vollzeit

Benefits dieser Stelle

Equity

Bonus

Benefits

Zusammenfassung

Shield AI is hiring a seasoned Data Engineer to design and operate end-to-end data pipelines in a Databricks lakehouse. You will own Bronze-layer ingestion, build Silver-layer transformations, and define reusable patterns with strong governance across domains.

You will implement platform constructs for secure delivery, build CI/CD for data assets, and ensure data quality, lineage, and compliance in a fast-moving, regulated environment.

Qualifikationen

• 12+ years of data engineering experience with production data pipelines.
• Deep Databricks experience including Delta Lake, Databricks Workflows, Spark with PySpark/Spark SQL.
• Working knowledge of Unity Catalog catalogs, schemas, tables, lineage, and access control concepts.
• Experience with batch, CDC, and streaming ingestion patterns and the operational trade-offs.
• Experience with CI/CD and deployment automation for data pipelines and platform assets.
• Strong SQL skills and solid grounding in data modeling fundamentals.
• Ability to understand the business and regulatory context of the data being processed.
• Experience applying data classification, segregation, security, retention, or compliance requirements in data engineering workflows.

Aufgaben

• Design and build ingestion pipelines from enterprise source systems into the Databricks lakehouse.
• Own Bronze-layer ingestion including raw landing patterns, metadata capture, load traceability, and recoverable ingestion design.
• Build Silver-layer pipelines for cleansing, standardization, deduplication, conformance, and quality enforcement.
• Define and evolve reusable ingestion and transformation patterns, templates, and engineering standards for onboarding domains.
• Implement and maintain Databricks platform constructs for secure delivery, including catalogs, schemas, service principals, and job orchestration.
• Build and maintain CI/CD pipelines for data platform assets, including ingestion code and tests across environments.
• Apply data classification, segregation, and handling requirements within pipeline design.
• Build data quality controls that reflect business meaning, record integrity, completeness, and domain behavior.
• Quarantine, flag, and route problematic records according to quality and compliance rules.
• Maintain documentation for source objects, ingestion logic, transformations, data quality rules, and limitations.
• Partner with Analytics Engineer and domain teams to ensure Silver/gold data reliability and governance.
• Collaborate with domain teams to align ownership boundaries, onboarding patterns, and data contracts.

Kenntnisse

Databricks

PySpark

Spark SQL

Unity Catalog

SQL

Data Modeling

Streaming

Batch processing

CDC

CI/CD

Deployment automation

Regulated environment

12+ years data engineering

Databricks Workflows

Ausbildung

Bachelor's or Master’s in a technical field

Tools

Databricks Workflows

Jobbeschreibung

Role Summary

Shield AI is a venture-backed defense-tech company with the mission of protecting service members and civilians with intelligent systems. Its products include Hivemind autonomy software, V-BAT and X-BAT aircraft, and Aechelon simulation and synthetic reality technologies. With offices and facilities across the U.S., Europe, the Middle East, and Asia-Pacific, Shield AI’s technology actively supports operations worldwide. For more information, visit www.shield.ai https://shield.ai/. Follow Shield AI on LinkedIn https://www.linkedin.com/company/shield-ai, X https://x.com/shieldaitech, Instagram https://www.instagram.com/shield_ai, and YouTube https://www.youtube.com/@Shield_AI.

What You’ll Do:

• Design and build ingestion pipelines, batch and streaming where appropriate, from enterprise source systems into the Databricks lakehouse using Delta Lake.
• Own Bronze-layer ingestion, including raw landing patterns, metadata capture, load traceability, and recoverable ingestion design.
• Build Silver-layer pipelines for cleansing, standardization, deduplication, conformance, and quality enforcement without embedding unauthorized or undocumented KPI logic.
• Define and evolve reusable ingestion and transformation patterns, templates, and engineering standards that other domains can adopt as they onboard to the platform.
• Implement and maintain Databricks platform constructs needed for secure delivery, including catalogs, schemas, service principals, job orchestration, and environment-aware deployment patterns.
• Build and maintain CI/CD pipelines for data platform assets, including ingestion code, transformation logic, workflow definitions, tests, and environment promotion across dev, test, and prod.
• Apply data classification, segregation, and handling requirements within the pipeline design, ensuring sensitive and regulated data is processed in accordance with enterprise controls and access policies.
• Build data quality controls that do more than detect technical failures, including checks that reflect actual business meaning, record integrity, completeness, and expected domain behavior.
• Quarantine, flag, and route problematic records or datasets according to defined quality and compliance rules rather than silently dropping or obscuring issues.
• Maintain documentation for source objects, ingestion logic, applied transformations, data quality rules, and known limitations so downstream teams can trust and use the data correctly.
• Partner with the Analytics Engineer and domain teams to ensure Silver-layer data is reliable, well-governed, and suitable for trusted Gold-layer modeling.
• Collaborate with domain engineering teams to align on ownership boundaries, onboarding patterns, data contracts, and support expectations as new domains are enabled onto the platform.

Preferred Qualifications:

• Databricks certification such as Data Engineer Associate or Professional.
• Experience standing up or maturing Unity Catalog structures and access models in an enterprise setting.
• Experience with CI/CD, infrastructure-as-code, and deployment automation for data platforms.
• Experience in defense, aerospace, financial services, healthcare, or another highly regulated environment.
• Experience enabling multiple business domains on a shared data platform while maintaining strong governance and ownership boundaries.

Required Qualifications:

• 12+ years of data engineering experience, including hands‑on ownership of production data pipelines.
• Strong Databricks experience, including Delta Lake, Databricks Workflows or Jobs, and Spark with PySpark and/or Spark SQL.
• Working knowledge of Unity Catalog, including catalogs, schemas, tables, lineage, and access control concepts.
• Experience with batch, CDC, and/or streaming ingestion patterns and the operational trade‑offs associated with each.
• Experience with CI/CD and deployment automation for data pipelines and platform assets, including version control, testing, and controlled promotion across environments.
• Strong SQL skills and solid grounding in data modelling fundamentals, even if dimensional modelling is not the primary responsibility of this role.
• Demonstrated ability to understand the business and regulatory context of the data being processed, not just the mechanics of pipeline development.
• Experience applying data classification, segregation, security, retention, or compliance requirements in data engineering workflows within a regulated or security-sensitive environment.
• Ability to design pipelines with awareness of the actual data domains involved, including sensitivity, ownership, permitted use, and downstream impact.
• Comfort operating in a fast‑moving platform build where patterns are still being established and engineers are expected to shape standards, not just follow them.

#LI-KE1 #LE Full‑time regular employee offer package: Pay within range listed + Bonus + Benefits + Equity

Temporary employee offer package: Pay within range listed above + temporary benefits package (applicable after 60 days of employment) Salary compensation is influenced by a wide array of factors including but not limited to skill set, level of experience, licenses and certif