Senior Data Engineer — End-to-End Databricks Pipelines

Protective

United States

Remoto

USD 84.000 - 129.000

Tempo pieno

14 giorni+

Generatore di candidature

Distinguiti per questa posizione — genera un curriculum e una lettera di presentazione personalizzati in circa un minuto.

Supera i filtri ATS

Vantaggi offerti da questo lavoro

Health, dental, and vision insurance

Paid time away benefits

401(k) plan with company matching

ProHealth Rewards program

Descrizione del lavoro

Protective is seeking a Data Engineer to design, build, and operate production data pipelines on Voyager, our Databricks lakehouse on Azure. You will own end-to-end data products from ingestion through Gold, writing production-grade Python and SQL and enforcing data contracts for trusted datasets.

You will join a pod that collaborates with data architects, analysts, and product owners to translate requirements into scalable data solutions, with governance via Unity Catalog and secure credentials

Competenze

Bachelor’s degree in Computer Science, Information Systems, Engineering, or a related field; equivalent practical experience considered.
3+ years building and supporting production data pipelines in a cloud data platform environment.
Strong hands-on Python and SQL. Both are used daily and neither substitutes for the other.
Hands-on experience with Databricks or a comparable Spark-based lakehouse, including Delta Lake tables, MERGE, and incremental load patterns.
Practical understanding of medallion / multi-layer lakehouse design, and the judgment to say what belongs in Bronze versus Silver versus Gold.
Experience ingesting data from APIs, relational databases, files, or SaaS applications, including the incremental and state-management problems that come with it.
Working knowledge of dimensional modeling — grain, keys, facts and dimensions, slowly changing dimensions — and of ELT design patterns and data quality practice.
Experience with orchestration and scheduling using Dagster, Databricks Workflows, Airflow, Azure Data Factory, or similar.
Git-based source control, pull request review, automated testing, and CI/CD as normal practice — Azure DevOps or comparable.
Experience troubleshooting production data failures, performance bottlenecks, and source-system changes.
Experience with pipeline monitoring and alerting, and a working understanding of what a freshness or quality SLA means once real consumers depend on it.
Ability to explain technical designs and trade-offs to both technical and non-technical partners.

Mansioni

Design, develop, and maintain production data pipelines on Databricks using Python, SQL, Apache Spark, and Delta Lake.
Build Bronze-layer ingestion that reliably captures data from APIs, relational databases, flat files, cloud storage, and SaaS platforms — using dlt (dltHub) and Databricks-native ingestion where each fits — including incremental loading, pagination, watermarking, state management, and replay after failure.
Develop Silver-layer transformations in dbt and Python over Delta Lake that cleanse, standardize, type, deduplicate, validate, conform, and enrich data so that it is reusable across domains.
Create Gold-layer data products: dimensional models, slowly changing dimensions, fact and bridge tables, aggregates, and serving tables aligned to how consumers actually query.
Produce and maintain the curated datasets ML engineering trains and serves models from — feature and training tables that are versioned and reproducible, not one-off extracts.
Author and maintain data contracts using the Open Data Contract Standard (ODCS) — schema with real semantics, named owner, known consumers, quality rules, and freshness expectations — and assess backward compatibility before every change.
Implement data quality as code: uniqueness and not-null on keys at minimum, plus referential, accepted-value, freshness, and custom business-rule tests, surfaced to producers and consumers rather than buried in logs.
Orchestrate ingestion and transformation as assets in Dagster, deployed to Dagster Cloud, and operate what you build across development, branch, and production deployments — schedules and sensors, asset dependencies, backfills, and run observability.
Apply governance through Unity Catalog — catalogs, schemas, external locations, grants, row- and column-level security, and lineage — and handle credentials through Azure Key Vault rather than in code.
Implement incremental and merge-based processing with Delta Lake (MERGE, schema evolution, time travel, OPTIMIZE) and tune Spark jobs, table layouts, and compute for performance and cost.
Troubleshoot production failures, data-quality issues, source-system changes, and late-arriving or duplicate data — including backfills and recovery — and take part in the pod’s on-call rotation for the pipelines it owns, with root-cause analysis that closes the gap rather than reopening the ticket.
Build and maintain CI/CD for data assets in Azure DevOps — automated tests and CI checks on dlt, dbt, and Dagster changes, promotion from development through branch deployments to production, and releases that are repeatable and auditable.
Instrument what you own for observability: freshness, volume, quality, latency, and cost, with alerting tied to the SLAs and SLOs your contract commits to instead of depending on someone noticing.
Work inside the platform’s control expectations — least-privilege access, secrets in Azure Key Vault, change management through pull request and pipeline, and audit evidence that falls out of the deployment path rather than being reconstructed later.
Participate in code review and document architecture, runbooks, and data products so others can discover, trust, and reuse them.
Work with data architects, analysts, product owners, and business stakeholders to translate requirements into maintainable data solutions.