Senior Data Engineer: End-to-End Databricks Lakehouse

Protective Life; United States; On-site; USD 84,000 - 129,000; Full time

Benefits offered by this job

Health Insurance

Dental Insurance

Vision Insurance

401(k) Plan with Company Match

Paid Time Off

Job summary

Protective Life is seeking a Data Engineer to design, build, and operate production data pipelines on Voyager, our Databricks lakehouse on Azure. You will own end-to-end data flows, writing production‑grade Python and SQL, and enforcing data contracts for real consumers.

You will join a delivery pod responsible for data products end to end, collaborating with data architects, analysts, and stakeholders to drive governance, quality, and observability across the data pipeline lifecycle.

Qualifications

• Bachelor's degree in Computer Science, Information Systems, Engineering, or related field.
• 3+ years building and supporting production data pipelines in a cloud data platform.
• Strong Python and SQL; both used daily.
• Hands-on Databricks / Spark with Delta Lake, MERGE, and incremental loads.
• Experience with medallion/lakehouse design and data quality.

Responsibilities

• Design, develop, and maintain production data pipelines on Databricks using Python, SQL, Spark, and Delta Lake.
• Ingest data from APIs, databases, files, and SaaS with incremental loading and state management.
• Develop Silver-layer transformations in dbt and Python over Delta Lake for cleansed data.
• Create Gold-layer data products and serving tables aligned to consumer queries.
• Maintain ML feature and training tables with versioning and reproducibility.
• Author data contracts using ODCS with clear ownership and consumers.
• Implement data quality tests and observable metrics; raise alerts per SLAs.
• Orchestrate pipelines with Dagster, Dagster Cloud, and CI/CD in Azure DevOps.
• Govern with Unity Catalog and secure credentials via Azure Key Vault.
• Troubleshoot production issues, backfills, and on-call incidents.

Skills

Python

SQL

Databricks

Delta Lake

dbt

Dagster

Airflow

ELT

Education

Bachelor's degree in Computer Science / Information Systems / Engineering

Tools

Dagster Cloud

Azure DevOps

Airflow

Delta Lake

Job description

Protective Life is seeking a Data Engineer to design, build, and operate production data pipelines on Voyager, our Databricks lakehouse on Azure. You will own end-to-end data flows, writing production‑grade Python and SQL, and enforcing data contracts for real consumers.

You will join a delivery pod responsible for data products end to end, collaborating with data architects, analysts, and stakeholders to drive governance, quality, and observability across the data pipeline lifecycle.