Senior Data Engineer — Databricks Lakehouse Lead

Remote; San Francisco (CA); On-site; USD 150,000 - 210,000; Full time

Benefits offered by this job

Health insurance

Paid time off

Parental leave

401(k) plan with Company matching

ProHealth Rewards

Job summary

Protective is seeking a Data Engineer to design, build, and operate production data pipelines on our Databricks lakehouse on Azure. You’ll own end-to-end data flow from Bronze to Gold, writing production-grade Python and SQL, enforcing data contracts, and treating data as a product with named owners and consumers.

You will be embedded on a delivery pod, collaborating with data architects, analysts, and product owners to deliver curated datasets for ML training and inference, while maintaining

Qualifications

• 3+ years building and supporting production data pipelines in a cloud data platform environment.
• Strong hands-on Python and SQL. Both are used daily and neither substitutes for the other.
• Hands-on experience with Databricks or a comparable Spark-based lakehouse, including Delta Lake tables, MERGE, and incremental load patterns.
• Practical understanding of medallion / multi-layer lakehouse design, and the judgment to say what belongs in Bronze versus Silver versus Gold.
• Experience ingesting data from APIs, relational databases, files, or SaaS applications, including incremental and state-management problems.
• Working knowledge of dimensional modeling — grain, keys, facts and dimensions, slowly changing dimensions — and ELT design patterns and data quality practice.
• Experience with orchestration and scheduling using Dagster, Databricks Workflows, Airflow, Azure Data Factory, or similar.
• Git-based source control, pull request review, automated testing, and CI/CD as normal practice — Azure DevOps or comparable.
• Experience troubleshooting production data failures, performance bottlenecks, and source-system changes.
• Experience with pipeline monitoring and alerting, and understanding of freshness/quality SLAs.

Responsibilities

• Design, develop, and maintain production data pipelines on Databricks using Python, SQL, Spark, and Delta Lake.
• Build Bronze-layer ingestion from APIs, relational databases, files, cloud storage, and SaaS platforms with incremental loading and state management.
• Develop Silver-layer transformations in dbt and Python over Delta Lake to cleanse, standardize, and enrich data.
• Create Gold-layer data products: dimensional models, slowly changing dimensions, and serving tables.
• Produce and maintain curated datasets ML engineering trains and serves models from with versioning and reproducibility.
• Author and maintain data contracts using the Open Data Contract Standard (ODCS) and assess backward compatibility before changes.
• Implement data quality as code with tests for keys, referential integrity, value checks, freshness, and business rules.
• Orchestrate ingestion and transformation as Dagster assets deployed to Dagster Cloud with schedules, sensors, and backfills.
• Apply governance through Unity Catalog and handle credentials via Azure Key Vault.
• Implement incremental and merge-based processing with Delta Lake and tune Spark for performance & cost.
• Troubleshoot production data issues, participate in on-call rotations, and perform root-cause analysis.
• Build and maintain CI/CD for data assets in Azure DevOps with tests and promotions.
• Instrument observability: monitor freshness, volume, quality, latency, and cost with SLAs/SLOs.
• Adhere to least-privilege access, secret management, and audit trails in deployments.
• Participate in code reviews and document architecture and data products.
• Collaborate with data architects, analysts, product owners, and stakeholders to translate requirements into data solutions.

Skills

Python

SQL

Databricks

Delta Lake

APIs

dbt

Dagster

Airflow

CI/CD

Git

Spark

Unity Catalog

Azure Data Factory

dltHub

Meltano

Education

Bachelor’s degree in Computer Science, Information Systems, Engineering, or related field

Tools

Databricks

dbt

Dagster

Airflow

Azure Data Factory

Unity Catalog

dltHub

Job description

Protective is seeking a Data Engineer to design, build, and operate production data pipelines on our Databricks lakehouse on Azure. You’ll own end-to-end data flow from Bronze to Gold, writing production-grade Python and SQL, enforcing data contracts, and treating data as a product with named owners and consumers.

You will be embedded on a delivery pod, collaborating with data architects, analysts, and product owners to deliver curated datasets for ML training and inference, while maintaining