Senior Data Engineer — Build Trusted Lakehouse Pipelines

Protective; Nebraska; Hybrid; USD 84,000 - 129,000; Full time

Benefits offered by this job

Health insurance

401(k) plan with company matching

Paid time off

Mental health benefits

Parental leave

Job summary

Protective seeks a Data Platform Engineer to design, build, and maintain production data pipelines on Databricks in the US-based NE region. You will own Bronze to Gold layers, implement Delta Lake MERGE and incremental patterns, and collaborate with data scientists, analysts, and product teams to deliver trustworthy, cataloged data products.

You will deploy Dagster workflows, manage lineage via Unity Catalog, and enforce data contracts with ODCS.

Qualifications

• Bachelor’s degree or equivalent practical experience in a related field.
• 3+ years building and supporting production data pipelines in a cloud data platform environment.
• Strong hands-on Python and SQL; both are used daily.
• Hands-on experience with Databricks or similar Spark-based lakehouse, Delta Lake, MERGE, and incremental loads.
• Understanding of medallion/multi-layer lakehouse design and Bronze/Silver/Gold roles.
• Experience ingesting data from APIs, databases, files, or SaaS with incremental/state-management challenges.
• Knowledge of dimensional modeling, ELT patterns, and data quality practice.
• Experience with orchestration/scheduling (Dagster, Databricks Workflows, Airflow, ADF).
• Git-based source control, PR review, automated testing, and CI/CD.
• Experience troubleshooting production data failures and performance bottlenecks.
• Familiarity with pipeline monitoring, alerting, and SLAs/SLOs.
• Ability to explain technical designs to varied stakeholders.

Responsibilities

• Design, develop, and maintain production data pipelines on Databricks using Python, SQL, Spark, and Delta Lake.
• Ingest data from APIs, DBs, files, cloud storage, and SaaS with incremental loading and state management.
• Develop Silver-layer transformations in dbt and Python over Delta Lake to cleanse and standardize data.
• Create Gold-layer data products and serving tables aligned with consumption patterns.
• Produce and maintain curated datasets for ML training and serving with versioning.
• Write data contracts using ODCS with semantic schemas, owners, consumers, and quality rules.
• Implement data quality as code across keys, referential integrity, and freshness constraints.
• Orchestrate ingestion/transformation as Dagster assets with Dagster Cloud deployments.
• Apply governance via Unity Catalog and manage credentials with Azure Key Vault.
• Implement incremental processing with Delta Lake and optimize for performance and cost.
• Troubleshoot data issues and participate in on-call rotations; perform root-cause analysis.

Skills

Python

SQL

Databricks

Delta Lake

Dagster

dbt

Airflow

Azure DevOps

CI/CD

Open Data Contract Standard (ODCS)

Education

Bachelor’s degree in CS/IS/Engineering

Tools

dltHub

Dagster Cloud

Azure Key Vault

Job description

Protective seeks a Data Platform Engineer to design, build, and maintain production data pipelines on Databricks in the US-based NE region. You will own Bronze to Gold layers, implement Delta Lake MERGE and incremental patterns, and collaborate with data scientists, analysts, and product teams to deliver trustworthy, cataloged data products.

You will deploy Dagster workflows, manage lineage via Unity Catalog, and enforce data contracts with ODCS.