Senior Data Engineer: Databricks Lakehouse Pipelines

Protective Life; Nebraska; On-site; USD 84,000 - 129,000; Full time

Benefits offered by this job

Health, dental and vision insurance

401(k) with company matching

Paid time off and parental leave

Employee assistance program

Job summary

Protective Life is seeking a Data Engineer to design, build, and operate production data pipelines on Voyager, our Databricks lakehouse on Azure. You will own end-to-end data flow from ingestion to Gold layer, applying cleansing, validation, and conformance to deliver trusted data products.

The role sits at the intersection of engineering, analytics, and platform operations, writing production-grade Python and SQL.

Qualifications

• Bachelor’s degree in Computer Science, Information Systems, Engineering, or related field.
• 3+ years building and supporting production data pipelines in a cloud data platform.
• Strong Python and SQL; both are used daily.
• Hands-on Databricks or comparable Spark-based lakehouse experience.
• Experience with medallion/multi-layer lakehouse design (Bronze/Silver/Gold).
• Ingestion from APIs, databases, files, or SaaS apps with incremental loading.
• Knowledge of dimensional modeling and ELT design patterns.
• Experience with Dagster, Databricks Workflows, Airflow, ADF, or similar.
• Git-based source control, PRs, automated testing, and CI/CD.

Responsibilities

• Design, build, and maintain production data pipelines on Databricks using Python, SQL, Spark, and Delta Lake.
• Develop Bronze ingestion from APIs, databases, files, and SaaS with incremental loading and state mgmt.
• Create Silver transformations in dbt and Python to cleanse, standardize, and enrich data.
• Publish Gold-layer data products and serving tables aligned to consumer queries.
• Produce feature and training datasets for ML models with versioning and reproducibility.
• Author data contracts using ODCS with owners, consumers, and freshness expectations.
• Implement data quality checks and observability with dashboards and alerts.
• Orchestrate pipelines with Dagster, Dagster Cloud, and promote through CI/CD.
• Governance via Unity Catalog and credentials in Azure Key Vault.
• Respond to production issues, backfills, and contribute to on-call rotation.

Skills

Python

SQL

Databricks

Delta Lake

ETL/ELT

Dagster

CI/CD

Git

Spark

dbt

Education

Bachelor’s degree in Computer Science, Information Systems, Engineering

Tools

Databricks Workflows

Azure DevOps

Dagster Cloud

Azure Key Vault

Delta Lake tooling

Job description

Protective Life is seeking a Data Engineer to design, build, and operate production data pipelines on Voyager, our Databricks lakehouse on Azure. You will own end-to-end data flow from ingestion to Gold layer, applying cleansing, validation, and conformance to deliver trusted data products.

The role sits at the intersection of engineering, analytics, and platform operations, writing production-grade Python and SQL.