Senior Data Engineer — Databricks Lakehouse Pipelines

Protective Life; Birmingham (AL); On-site; USD 84,000 - 129,000; Full time

Benefits offered by this job

Health insurance

401(k) plan with Company matching

Paid time off

Job summary

Protective Life in Birmingham, AL seeks a Data Engineer to design, build, and operate production data pipelines on Databricks lakehouse on Azure. You will own end-to-end data products from Bronze to Gold, writing production-grade Python and SQL and enforcing data contracts.

You will collaborate with data scientists and analysts, implement data quality tests, and orchestrate pipelines with Dagster or Airflow, while ensuring governance via Unity Catalog and secure credentials in Azure Key Vault.

Qualifications

• Bachelor’s degree in Computer Science, Information Systems, Engineering, or related field.
• 3+ years building production data pipelines in cloud data platforms.
• Strong Python and SQL skills.
• Experience with Databricks or Spark-based lakehouse, Delta Lake, MERGE.
• Knowledge of medallion architecture (Bronze/Silver/Gold).
• Ingest data from APIs, databases, files, or SaaS apps.
• Orchestration with Dagster, Airflow, or Azure Data Factory.
• Git-based source control and CI/CD as normal practice.

Responsibilities

• Design, build, and maintain production data pipelines on Databricks using Python, SQL, Spark, and Delta Lake.
• Ingest data from APIs, databases, files, and SaaS platforms with incremental loading and state management.
• Develop Silver-layer transformations in dbt and Python to cleanse and conformance.
• Create Gold-layer data products and serving tables aligned to consumption patterns.
• Publish curated datasets for ML training and production use with versioning.
• Author data contracts using Open Data Contract Standard and enforce data quality tests.
• Orchestrate pipelines with Dagster or similar tools and monitor end-to-end data flows.
• Apply governance with Unity Catalog and Azure Key Vault for credentials.

Skills

Python

SQL

Databricks

Delta Lake

dbt

Dagster

Airflow

Azure

Education

Bachelor’s degree in Computer Science or related field

Tools

Azure Databricks

Azure DevOps

dbt

Job description

Protective Life in Birmingham, AL seeks a Data Engineer to design, build, and operate production data pipelines on Databricks lakehouse on Azure. You will own end-to-end data products from Bronze to Gold, writing production-grade Python and SQL and enforcing data contracts.

You will collaborate with data scientists and analysts, implement data quality tests, and orchestrate pipelines with Dagster or Airflow, while ensuring governance via Unity Catalog and secure credentials in Azure Key Vault.