Azure Databricks Data Engineer - Delta Live Tables Expert

Insight Global (United States)

United States

Teletrabalho

USD 140 000 - 190 000

Tempo integral

14 dias+

Recebe uma resposta deste empregador

O teu currículo e carta de apresentação, adaptados exatamente ao que estão a contratar.

Ultrapassa os filtros ATS

Resumo da oferta

Insight Global is hiring a senior data engineer for a 100% remote US role. You will design and deploy Azure Databricks data pipelines using PySpark and Delta Live Tables, modernizing on-prem SQL pipelines into scalable cloud-native solutions.

You will build configuration-driven pipelines with YAML/JSON, maintain CI/CD for data workflows, and implement Medallion Architecture across Bronze, Silver, and Gold. Prior healthcare experience is a plus.

Qualificações

  • 7+ years of data engineering experience building large-scale data platforms.
  • Strong hands-on experience with Azure Databricks, Delta Lake, and ADLS.
  • Advanced PySpark development experience, including DataFrames, transformations, joins, aggregations, and performance tuning.
  • Experience developing and maintaining Delta Live Tables (DLT) pipelines.
  • Strong SQL background with experience supporting legacy data warehouse environments.
  • Experience migrating SQL-based pipelines to modern Python/PySpark architectures.
  • Experience with Infrastructure as Code (IaC) and Pipeline as Code methodologies.
  • Proficiency with YAML, JSON, and configuration-driven development patterns.
  • Strong understanding of Medallion Architecture (Bronze, Silver, Gold).
  • Experience building ingestion, transformation, and curation pipelines across Bronze, Silver, and Gold layers.
  • Experience troubleshooting, monitoring, and supporting production data pipelines.
  • Familiarity with Databricks Asset Bundles, Unity Catalog, and Delta Lake preferred.
  • Healthcare industry experience preferred.
  • Medicare, payer, member enrollment, or claims data experience strongly preferred.
  • Understanding of healthcare data quality, PHI/PII handling, and regulatory requirements preferred.

Responsabilidades

  • Design, develop, and deploy Azure Databricks data pipelines using PySpark and Delta Live Tables.
  • Modernize legacy on-prem SQL pipelines into scalable cloud-native Databricks solutions.
  • Build configuration-driven data pipelines using YAML, JSON, and reusable framework patterns.
  • Develop and maintain CI/CD deployment processes for data engineering workflows.
  • Implement and support Medallion Architecture pipelines across Bronze, Silver, and Gold layers.
  • Partner with upstream and downstream teams to ensure data quality, reliability, and scalability.
  • Troubleshoot and resolve production pipeline issues across Databricks and legacy SQL environments.
  • Enhance monitoring, observability, and operational support processes.
  • Support Medicare and payer-focused initiatives involving member, claims, and operational datasets.
  • Follow established architecture standards and engineering best practices to ensure platform consistency and maintainability.

Conhecimentos

PySpark
SQL
Data pipelines
IaC
YAML/JSON
Configuration-driven
Healthcare data concepts

Ferramentas

Azure Databricks
Delta Lake
Delta Live Tables
ADLS
Unity Catalog
Databricks Asset Bundles

Descrição da oferta de emprego

Insight Global is hiring a senior data engineer for a 100% remote US role. You will design and deploy Azure Databricks data pipelines using PySpark and Delta Live Tables, modernizing on-prem SQL pipelines into scalable cloud-native solutions.

You will build configuration-driven pipelines with YAML/JSON, maintain CI/CD for data workflows, and implement Medallion Architecture across Bronze, Silver, and Gold. Prior healthcare experience is a plus.

Obtém a tua avaliação gratuita e confidencial do currículo.

ou arrasta e larga o ficheiro aqui.

Similar jobs

Ofertas semelhantes que vale a pena comparar

Senior Azure Databricks Engineer - Data Pipelines & Delta
Senior Azure Databricks Engineer - Data Pipelines & Delta

ITMC Systems, Inc • EUA

Teletrabalho
USD 140 000 - 190 000
Azure data engineer
Azure data engineer

Valce Talent Solutions • EUA

Teletrabalho
USD 49 000 - 81 000
Databricks Engineer
Databricks Engineer

Insight Global (United States) • EUA

Teletrabalho
USD 140 000 - 190 000
Azure Data Platform Engineer: Delta Lake & Spark Expert
Azure Data Platform Engineer: Delta Lake & Spark Expert

Techwish • New York (NY)

Presencial
USD 140 000 - 190 000
Remote Databricks Engineer — Spark & Delta Lake Expert
Remote Databricks Engineer — Spark & Delta Lake Expert

Arting Digital Private Limited • EUA

Teletrabalho
USD 120 000 - 170 000
Remote Databricks Engineer: Spark, Delta Lake & ETL
Remote Databricks Engineer: Spark, Delta Lake & ETL

Artingdigital • EUA

Teletrabalho
USD 120 000 - 180 000
Remote Databricks Engineer — Delta Lake & ETL Expert
Remote Databricks Engineer — Delta Lake & ETL Expert

Arting Digital • EUA

Teletrabalho
USD 120 000 - 180 000
Azure Data Engineer - Databricks, PySpark & Healthcare Data
Azure Data Engineer - Databricks, PySpark & Healthcare Data

Valiant Technologies LLC • Cupertino (CA)

Presencial
USD 120 000 - 170 000
Azure Data Engineer: Databricks & Spark Expert (Hybrid)
Azure Data Engineer: Databricks & Spark Expert (Hybrid)

Signature IT World • New Jersey

Híbrido
USD 110 000 - 150 000
Senior Data Engineer - Azure Pipelines (Remote)
Senior Data Engineer - Azure Pipelines (Remote)

Diverse Lynx, LLC • Minneapolis (MN)

Teletrabalho
USD 69 000 - 76 000