Principal Data Engineer: Lakehouse Migration & Data Quality

Horizontal Talent; Golden Valley (MN); On-site; USD 68,000 - 117,000; Full time

Benefits offered by this job

Medical, dental, vision benefits

Retirement plan

Job summary

Horizontal Talent is seeking a hands‑on Principal Data Engineer to re‑engineer an enterprise data platform and support migration from Azure Synapse Analytics to Databricks. You will strengthen data quality, governance and ensure accuracy of critical data flows during modernization.

You will review pipelines, data models and governance controls, validate end‑to‑end processing, and collaborate across governance partners to confirm rules and business needs.

Qualifications

• 7+ years hands-on data engineering across enterprise platforms or migrations.
• Strong experience with Azure Synapse Analytics including pipelines, Spark and dedicated SQL pools.
• Solid understanding of ADLS Gen2, Delta Lake, and lakehouse patterns.
• Hands-on Databricks, PySpark, Spark SQL, and Delta Lake features (MERGE, CDC, schema enforcement, time travel).
• Advanced Python and SQL for ETL/ELT at scale with governance and quality checks.
• Familiarity with SCD Type 0/1/2, deletes, and retroactive changes.

Responsibilities

• Assess and re‑engineer data platform components across ingestion, transformation, storage and consumption.
• Review pipeline logic, models, orchestration patterns and governance controls to identify gaps.
• Validate end‑to‑end processing including CDC, SCDs, deduplication and reconciliation.
• Support migration of workloads to Databricks lakehouse while preserving logic and controls.
• Build/refine pipelines with lakehouse patterns and ensure output consistency during cutover.
• Recreate ingestion/transformation/orchestration on target platforms with auditability and quality checks.
• Collaborate with business, architecture and governance to confirm rules and data requirements.
• Contribute to code reviews, mentoring and planning for legacy retirement.

Skills

Data engineering

Azure Synapse

Delta Lake

Databricks

PySpark

Python

SQL

Data governance

SCD patterns

Data quality

Tools

Unity Catalog

Terraform

ARM

Docker

Azure DevOps

Oracle GoldenGate

Job description

Horizontal Talent is seeking a hands‑on Principal Data Engineer to re‑engineer an enterprise data platform and support migration from Azure Synapse Analytics to Databricks. You will strengthen data quality, governance and ensure accuracy of critical data flows during modernization.

You will review pipelines, data models and governance controls, validate end‑to‑end processing, and collaborate across governance partners to confirm rules and business needs.