Data Engineer, Principal

Horizontal Talent; Golden Valley (MN); On-site; USD 68,000 - 117,000; Full time

Benefits offered by this job

Medical, dental, vision benefits

Retirement plan

Job summary

Horizontal Talent is seeking a hands‑on Principal Data Engineer to re‑engineer an enterprise data platform and support migration from Azure Synapse Analytics to Databricks. You will strengthen data quality, governance and ensure accuracy of critical data flows during modernization.

You will review pipelines, data models and governance controls, validate end‑to‑end processing, and collaborate across governance partners to confirm rules and business needs.

Qualifications

• 7+ years hands-on data engineering across enterprise platforms or migrations.
• Strong experience with Azure Synapse Analytics including pipelines, Spark and dedicated SQL pools.
• Solid understanding of ADLS Gen2, Delta Lake, and lakehouse patterns.
• Hands-on Databricks, PySpark, Spark SQL, and Delta Lake features (MERGE, CDC, schema enforcement, time travel).
• Advanced Python and SQL for ETL/ELT at scale with governance and quality checks.
• Familiarity with SCD Type 0/1/2, deletes, and retroactive changes.

Responsibilities

• Assess and re‑engineer data platform components across ingestion, transformation, storage and consumption.
• Review pipeline logic, models, orchestration patterns and governance controls to identify gaps.
• Validate end‑to‑end processing including CDC, SCDs, deduplication and reconciliation.
• Support migration of workloads to Databricks lakehouse while preserving logic and controls.
• Build/refine pipelines with lakehouse patterns and ensure output consistency during cutover.
• Recreate ingestion/transformation/orchestration on target platforms with auditability and quality checks.
• Collaborate with business, architecture and governance to confirm rules and data requirements.
• Contribute to code reviews, mentoring and planning for legacy retirement.

Skills

Data engineering

Azure Synapse

Delta Lake

Databricks

PySpark

Python

SQL

Data governance

SCD patterns

Data quality

Tools

Unity Catalog

Terraform

ARM

Docker

Azure DevOps

Oracle GoldenGate

Job description

We are seeking a hands‑on Principal Data Engineer to help re‑engineer an enterprise data platform and support the migration of validated workloads from Azure Synapse Analytics to Databricks. This role is ideal for someone who enjoys reverse‑engineering complex solutions, strengthening data quality and governance, and ensuring business‑critical data flows remain accurate throughout modernization.

Responsibilities

• Assess and re‑engineer existing data platform components across ingestion, transformation, storage, and consumption layers.
• Review current pipeline logic, data models, orchestration patterns, and governance controls to identify gaps or technical debt.
• Validate end‑to‑end data processing behavior, including change data capture, slowly changing dimensions, deduplication, and reconciliation routines.
• Support the migration of workloads to a Databricks lakehouse environment while preserving business logic and control framework behavior.
• Build and refine pipelines using modern lakehouse patterns and ensure output consistency during parallel‑run and cutover activities.
• Recreate ingestion, transformation, and orchestration solutions on target platforms with strong auditability and quality checks.
• Collaborate with business, architecture, and governance partners to confirm rules, controls, and data requirements.
• Contribute to code reviews, technical mentoring, and planning for legacy component retirement.

Skills

• 7+ years of hands‑on data engineering experience, including work on complex enterprise platforms or migration initiatives.
• Strong experience with Azure Synapse Analytics, including pipelines, Spark, and dedicated SQL pools.
• Solid understanding of ADLS Gen2, Delta Lake, and modern lakehouse architecture patterns.
• Hands‑on knowledge of Databricks, PySpark, Spark SQL, and Delta Lake capabilities such as MERGE, CDC, schema enforcement, and time travel.
• Advanced Python and SQL skills for building and supporting ETL/ELT solutions at scale.
• Deep familiarity with data governance, audit, reconciliation, control frameworks, and data quality validation.
• Experience working with enterprise data patterns such as SCD Type 0/1/2, soft and hard deletes, and retroactive changes.
• Ability to analyze existing implementations, determine intent, and make sound re‑engineering decisions.
• Clear communication skills and the ability to work effectively with technical and non‑technical stakeholders.

Preferred Skills

• Experience with Azure Data Lake Storage Gen2 and Oracle GoldenGate for real‑time integration.
• Exposure to Azure Analysis Services and Power BI consumption‑layer patterns.
• Experience implementing governance features in Databricks, including Unity Catalog and audit logging.
• Familiarity with Azure DevOps, GitHub Actions, Terraform, ARM, or Docker.
• Knowledge of automated testing approaches for data pipelines, including reconciliation‑based validation.
• Experience integrating sources such as Oracle, SQL Server, Azure SQL Database, file systems, Salesforce, or APIs.
• Exposure to Azure Purview, monitoring tools such as Dynatrace, or streaming technologies such as Event Hub, Kafka, or Kinesis.
• Background in regulated or enterprise‑scale environments.

Horizontal is committed to building an inclusive workplace where different perspectives, experiences, and approaches are valued. We welcome candidates who share our commitment to diversity, equity, and inclusion and who are excited to contribute in a collaborative, respectful environment.

For those that join the team, we offer competitive compensation and benefits including medical, dental, vision, and retirement.

The pay range for this role is $49 - $85 per hour based on qualifications and experience.