Lead Data Engineer

OpenKrill; United States; On-site; USD 170,000 - 235,000; Full time

Job summary

Protective is seeking a Lead Data Engineer to set the technical direction for a data delivery pod building data products on Voyager, our Databricks lakehouse on Azure. You will design end-to-end data products and own the governance, contracts, and observability that power trusted analytics for customers.

You’ll lead design decisions on Bronze-Silver-Gold layers, enforce data contracts, and drive quality through tests and CI/CD, while still coding daily and mentoring engineers on the team.

Qualifications

• Bachelor’s degree in CS, IS, engineering, or related field; practical experience accepted.
• 6+ years building and operating production data pipelines and consumer-facing data models.
• Strong Python and SQL; able to design and review production code.
• Hands-on Databricks or Spark-based lakehouse experience (Delta Lake, MERGE, incremental processing, performance tuning).
• Deep dimensional modeling experience—grain, keys, slowly changing dimensions, facts and dimensions.
• Proven technical leadership: setting standards and guiding other engineers.
• Experience owning data that other teams depend on, including handling breaking changes and production incidents.
• Experience with orchestration (Dagster, Databricks Workflows, Airflow), Git-based development, CI/CD — Azure DevOps or comparable.
• Experience setting observability and SLA/SLO expectations for data others depend on, and running incident communication.
• Ability to explain trade-offs clearly to engineers, product owners, and stakeholders.

Responsibilities

• Lead the end-to-end design of data products: ingestion, cleansing, conformance, and Gold layer.
• Own dimensional design: grain, keys, Type 2 history, facts, bridges, and conformed dimensions.
• Decide where logic belongs across Silver, Gold, and consumer concerns.
• Keep models simple and robust to the questions asked.
• Partner with ML engineering when Gold is training data source; contracts and versioning.
• Define and enforce data contracts with owners and consumers.
• Set engineering standards for Python, SQL, dbt, testing, and repos; align with CI gates.
• Lead code reviews and ensure quality via tests, observability, and pipeline health.
• Own operational posture: backfills, on-call, incident analysis, and runbooks.
• Ensure secure change management and audit evidence from CI/CD.
• Develop reusable patterns to raise consistency and speed.
• Mentor engineers and share knowledge to avoid single points of failure.
• Collaborate with platform and governance on Unity Catalog and access.

Skills

Python

SQL

Databricks / Delta Lake

Dimensional modeling

Data pipelines

Technical leadership

Orchestration (Dagster, Airflow)

Git-based CI/CD

Azure DevOps

Observability / SLAs/SLOs

Communicating trade-offs

Education

Bachelor’s degree in Computer Science, Information Systems, Engineering, or related field

Tools

Databricks

Azure DevOps

Dagster

Airflow

Git

Delta Lake

Job description

The work we do has an impact on millions of lives, and you can be a part of it.We help protect our customers against life’s uncertainties. Regardless of where you work within the company, you’ll be helping provide protection and peace of mind when our customers need it most.

Protective is looking for a Lead Data Engineer to set the technical direction for a delivery pod building data products on Voyager, our Databricks lakehouse on Azure. You will lead the design of data products through the full medallion architecture - Bronze ingestion, Silver conformance, and Gold consumption - and be accountable for whether those products hold up for the consumers who depend on them.

This is a hands‑on technical leadership role, not a people‑management or project‑management role. You will still write and review production code daily. What you own is how the pod's data products are designed, modeled, contracted, and tested, and the standard the pod holds itself to. The Product Owner owns the backlog and the Scrum Master owns the sprint; you own the engineering.

On Voyager, the medallion layers are named Raw, Prep, and Prod. They map directly to Bronze, Silver, and Gold and are used interchangeably in this description.

Key ResponsibilitiesDesign and modeling

• Lead the design of data products end to end: what gets ingested, how it is cleansed and conformed, how it is modeled, and what the Gold layer looks like to the people querying it.
• Own dimensional design — grain, natural and surrogate keys, Type 2 history, facts, bridges, and conformed dimensions shared across the pod’s products.
• Set the pod’s position on where logic belongs: what is cleaned in Silver, what is business logic in Gold, and what is a consumer’s own concern.
• Keep models as simple as the questions require, and push back on designs that will not hold.
• Partner with ML engineering where the pod’s Gold layer is the training or feature source for a model, so those datasets are contracted, versioned, and reproducible like any other consumer‑facing product.

Data contracts and consumer compatibility

• Own the pod’s ODCS data contracts as real interfaces: named owners, named consumers, enforceable quality rules, freshness and update expectations, and an explicit breaking‑change policy.
• Make the compatibility call on every proposed contract change, and drive consumer notification when a change is genuinely breaking.
• Represent the pod’s contracts in cross‑domain conversations, where one pod’s Gold layer is another team’s dependency.

Standards, quality, and operations

• Set and hold the pod’s engineering standards for Python, SQL, dbt, testing, model structure, naming, and repository conventions — consistent with the platform’s paved paths and Azure DevOps CI gates rather than in competition with them.
• Lead code review. Be the reviewer who catches the modeling mistake, the missing test, and the change that will break a consumer, and who explains why so the pod learns it.
• Ensure quality rules are enforced in tests and asset checks rather than asserted in documentation, and that pipeline health is observable without someone going to look — freshness, volume, latency, and cost instrumented, and alerting set against the SLAs and SLOs the pod’s contracts commit to.
• Own the pod’s operational posture for its own pipelines: failure diagnosis, data‑issue triage, backfills, cost and performance tuning, on‑call coverage and escalation, root‑cause analysis, and runbooks someone other than the author can execute.
• Keep the pod’s delivery inside the control expectations of a regulated carrier: change management through pull request and pipeline, segregation of duties between authoring and deploying, least‑privilege access, and audit evidence that falls out of CI/CD rather than being assembled for an auditor.
• Develop reusable frameworks, templates, and patterns that raise the pod’s consistency and delivery speed.

Delivery leadership

• Partner with the Product Owner and Scrum Master on decomposition and refinement: turn use cases and features into estimable, grounded stories with testable acceptance criteria and an identified target layer and repository.
• Hold the Definition of Ready before the pod commits and the Definition of Done before the pod calls something finished — merged and approved code, passing CI and coverage gates, and evidence that the outcome is real.
• Identify unknowns that need a spike rather than an estimate, and say so during planning rather than mid‑sprint.

Mentorship and collaboration

• Grow the engineers on the pod through design review, pairing, and code review rather than by taking the hard work yourself.
• Bring new engineers up to productive speed on the platform’s conventions and tooling, and reduce single points of knowledge — no data product that only one person understands.
• Work with the platform team on capability gaps: when the pod needs something the platform does not yet offer, raise it as a demand signal rather than building a private workaround.
• Partner with the DataOps/MLOps Lead on the shared CI/CD, orchestration, and observability standards — adopt and strengthen the paved path rather than forking it, and be the pod’s voice on what it is still missing.
• Work with data architecture and governance on solution shape, Unity Catalog placement, and access requirements.

QualificationsRequired Qualifications

• Bachelor’s degree in Computer Science, Information Systems, Engineering, or a related field; equivalent practical experience considered.
• 6+ years building and operating production data pipelines and consumer‑facing data models, from source ingestion through to published data products.
• Strong hands‑on Python and SQL, with the credibility to make design calls and the willingness to still write and review code.
• Hands‑on experience with Databricks or a comparable Spark‑based lakehouse, including Delta Lake, MERGE, incremental processing, and performance tuning.
• Deep dimensional modeling experience — grain, keys, slowly changing dimensions, facts and dimensions, conformed dimensions.
• Demonstrated technical leadership: setting standards, leading design, and raising other engineers’ work, whether or not the role carried a lead title.
• Experience owning data that other teams depend on, including handling breaking changes and production data incidents.
• Experience with orchestration (Dagster, Databricks Workflows, Airflow, or similar), Git‑based collaborative development, code review, and CI/CD — Azure DevOps or comparable.
• Experience setting observability and SLA/SLO expectations for data other teams depend on, and running the incident and communication path when they are missed.
• Ability to explain trade‑offs clearly to engineers, product owners, and business stakeholders, and to say no to a design that will not hold.

Preferred Qualifications

• Databricks certification (Data Engineer Professional or equivalent demonstrated depth).
• Unity Catalog at multi‑team scale: catalogs, schemas, external locations, permissions, and lineage.
• dbt at scale on Databricks, and Python‑based modeling frameworks over Delta Lake.
• Dagster and Dagster Cloud, including assets, asset checks, and branch deployments.
• Practical experience with data contracts, ODCS, or data‑mesh style data product ownership.
• Experience with a declarative Python ingestion framework such as dlt (dltHub) or comparable.
• Data quality and observability tooling such as Great Expectations, Monte Carlo, or similar.
• Familiarity with MLOps practice — MLflow, model registries, and model serving — sufficient to design data products that ML systems can depend on.
• Azure and Azure DevOps.
• Financial services, insurance, or another regulated industry, including data access, lineage, and audit expectations.
• Experience introducing AI‑assisted development into a team’s normal workflow in a disciplined way.

Employee Benefits:

We aim to protect the wellbeing of our employees and