Senior Azure Databricks Data Engineer: Medallion Pipelines

Insight Global; Woonsocket (RI); On-site; USD 120,000 - 160,000; Full time

Job summary

Insight Global is seeking a senior Data Engineer in the United States to design, build, and deploy Azure Databricks pipelines using PySpark and Delta Live Tables. You will modernize legacy SQL pipelines into scalable cloud-native Databricks solutions and develop configuration-driven data pipelines with YAML/JSON patterns.

You will also implement CI/CD for data workflows, support Medallion Architecture across Bronze/Silver/Gold, and ensure data quality with upstream/downstream partners.

Qualifications

• 5-7+ years of Data Engineering experience building large-scale data platforms.
• Strong hands-on experience with Azure Databricks, Delta Lake, and ADLS.
• Advanced PySpark development and SQL knowledge, incl. performance tuning.

Responsibilities

• Design, develop, and deploy Azure Databricks data pipelines using PySpark and Delta Live Tables.
• Modernize legacy on-prem SQL pipelines into scalable cloud-native Databricks solutions.
• Build configuration-driven data pipelines using YAML, JSON, and reusable framework patterns.
• Develop and maintain CI/CD deployment processes for data engineering workflows.
• Implement and support Medallion Architecture pipelines across Bronze, Silver, and Gold layers.
• Partner with upstream and downstream teams to ensure data quality, reliability, and scalability.
• Troubleshoot and resolve production pipeline issues across Databricks and legacy SQL environments.
• Enhance monitoring, observability, and operational support processes.
• Support Medicare and payer-focused initiatives involving member, claims, and operational datasets.
• Follow established architecture standards and engineering best practices to ensure platform consistency and maintainability.

Skills

Data Engineering

Python / PySpark

SQL

Cloud data platforms

Tools

Azure Databricks

Delta Lake

Azure Data Lake Storage

Delta Live Tables

Unity Catalog

Databricks Asset Bundles

IaC (Terraform/ARM)

YAML/JSON

Job description

Insight Global is seeking a senior Data Engineer in the United States to design, build, and deploy Azure Databricks pipelines using PySpark and Delta Live Tables. You will modernize legacy SQL pipelines into scalable cloud-native Databricks solutions and develop configuration-driven data pipelines with YAML/JSON patterns.

You will also implement CI/CD for data workflows, support Medallion Architecture across Bronze/Silver/Gold, and ensure data quality with upstream/downstream partners.