Lead Data Engineer - Architect Data Products on Databricks

Openkrill; United States; Remote; USD 170,000 - 235,000; Full time

Job summary

Protective is seeking a Lead Data Engineer to set the technical direction for a data delivery pod building data products on Voyager, our Databricks lakehouse on Azure. You will design end-to-end data products and own the governance, contracts, and observability that power trusted analytics for customers.

You’ll lead design decisions on Bronze-Silver-Gold layers, enforce data contracts, and drive quality through tests and CI/CD, while still coding daily and mentoring engineers on the team.

Qualifications

• Bachelor’s degree in CS, IS, engineering, or related field; practical experience accepted.
• 6+ years building and operating production data pipelines and consumer-facing data models.
• Strong Python and SQL; able to design and review production code.
• Hands-on Databricks or Spark-based lakehouse experience (Delta Lake, MERGE, incremental processing, performance tuning).
• Deep dimensional modeling experience—grain, keys, slowly changing dimensions, facts and dimensions.
• Proven technical leadership: setting standards and guiding other engineers.
• Experience owning data that other teams depend on, including handling breaking changes and production incidents.
• Experience with orchestration (Dagster, Databricks Workflows, Airflow), Git-based development, CI/CD — Azure DevOps or comparable.
• Experience setting observability and SLA/SLO expectations for data others depend on, and running incident communication.
• Ability to explain trade-offs clearly to engineers, product owners, and stakeholders.

Responsibilities

• Lead the end-to-end design of data products: ingestion, cleansing, conformance, and Gold layer.
• Own dimensional design: grain, keys, Type 2 history, facts, bridges, and conformed dimensions.
• Decide where logic belongs across Silver, Gold, and consumer concerns.
• Keep models simple and robust to the questions asked.
• Partner with ML engineering when Gold is training data source; contracts and versioning.
• Define and enforce data contracts with owners and consumers.
• Set engineering standards for Python, SQL, dbt, testing, and repos; align with CI gates.
• Lead code reviews and ensure quality via tests, observability, and pipeline health.
• Own operational posture: backfills, on-call, incident analysis, and runbooks.
• Ensure secure change management and audit evidence from CI/CD.
• Develop reusable patterns to raise consistency and speed.
• Mentor engineers and share knowledge to avoid single points of failure.
• Collaborate with platform and governance on Unity Catalog and access.

Skills

Python

SQL

Databricks / Delta Lake

Dimensional modeling

Data pipelines

Technical leadership

Orchestration (Dagster, Airflow)

Git-based CI/CD

Azure DevOps

Observability / SLAs/SLOs

Communicating trade-offs

Education

Bachelor’s degree in Computer Science, Information Systems, Engineering, or related field

Tools

Databricks

Azure DevOps

Dagster

Airflow

Git

Delta Lake

Job description

Protective is seeking a Lead Data Engineer to set the technical direction for a data delivery pod building data products on Voyager, our Databricks lakehouse on Azure. You will design end-to-end data products and own the governance, contracts, and observability that power trusted analytics for customers.

You’ll lead design decisions on Bronze-Silver-Gold layers, enforce data contracts, and drive quality through tests and CI/CD, while still coding daily and mentoring engineers on the team.