Azure Databricks Engineer

Insight Global; Woonsocket (RI); On-site; USD 120,000 - 160,000; Full time

Job summary

Insight Global is seeking an experienced Azure Databricks Data Engineer to modernize healthcare data platforms, designing, building, and maintaining scalable pipelines in Azure Databricks while migrating legacy SQL workloads to a cloud-native architecture.

The role emphasizes Pipeline as Code with YAML/JSON, automated CI/CD, Medallion Architecture, and governance, with a focus on healthcare data environments such as Medicare and payer datasets.

Qualifications

• 5+ years of Data Engineering experience building large-scale data platforms.
• Strong hands-on experience with Azure Databricks, Delta Lake, and ADLS.
• Advanced PySpark development experience including DataFrames, transformations, joins, aggregations, and performance tuning.

Responsibilities

• Design, develop, and deploy new Azure Databricks data pipelines using PySpark and Delta Live Tables
• Translate and modernize legacy on-prem SQL pipelines into scalable cloud-native Databricks solutions
• Build configuration-driven data pipelines utilizing YAML, JSON, and reusable framework patterns
• Develop and maintain automated CI/CD deployment processes for data engineering workflows
• Implement and support Medallion Architecture data pipelines across Bronze, Silver, and Gold layers
• Partner with upstream and downstream teams to ensure data quality, reliability, and platform scalability
• Troubleshoot and resolve production pipeline failures within both Databricks and legacy SQL environments
• Enhance monitoring, observability, and operational support processes for healthcare data pipelines
• Support Medicare and payer-focused data initiatives, including member, claims, and operational datasets

Skills

Azure Databricks

PySpark

Delta Live Tables

Medallion Architecture

Pipeline as Code

YAML/JSON

CI/CD

SQL

Healthcare data

Tools

Delta Lake

ADLS

Unity Catalog

Databricks Asset Bundles

Job description

Job Description

Job Description

We are seeking an experienced Azure Databricks Data Engineer to support the modernization and enhancement of enterprise healthcare data platforms. This role will focus on designing, building, and maintaining scalable data pipelines within Azure Databricks while supporting the migration of legacy SQL-based workloads to a modern cloud architecture. Approximately 70% of the role will involve new development and pipeline engineering, while 30% will focus on production support, troubleshooting, and remediation of existing data pipelines.

The ideal candidate will have strong expertise in Azure Databricks, PySpark, Delta Live Tables (DLT), Medallion Architecture, and Infrastructure as Code / Pipeline as Code concepts leveraging YAML and configuration-driven development. Experience supporting Medicare, payer, or claims-based healthcare data environments is highly preferred. The role aligns closely with a configuration-driven, "Pipeline as Code" architecture built around YAML configurations, reusable frameworks, and automated CI/CD deployments.

We are a company committed to creating diverse and inclusive environments where people can bring their full, authentic selves to work every day. We are an equal opportunity/affirmative action employer that believes everyone matters. Qualified candidates will receive consideration for employment regardless of their race, color, ethnicity, religion, sex (including pregnancy), sexual orientation, gender identity and expression, marital status, national origin, ancestry, genetic factors, age, disability, protected veteran status, military or uniformed service member status, or any other status or characteristic protected by applicable laws, regulations, and ordinances. If you need assistance and/or a reasonable accommodation due to a disability during the application or recruiting process, please send a request to [e-mail] learn more about how we collect, keep, and process your private information, please review Insight Global's Workforce Privacy Policy: https://insightglobal.com/workforce-privacy-policy/.

Skills and Requirements

5+ years of Data Engineering experience building large-scale data platforms

Strong hands-on experience with Azure Databricks, Delta Lake, and ADLS

Advanced PySpark development experience including DataFrames, transformations, joins, aggregations, and performance tuning

Experience developing and maintaining Delta Live Tables (DLT) pipelines

Strong SQL experience and background supporting legacy data warehouse environments

Experience migrating SQL-based pipelines into modern Python/PySpark-based architectures

Experience with Infrastructure as Code (IaC) and Pipeline as Code methodologies

Proficiency working with YAML, JSON, and configuration-driven development patterns

Experience automating CI/CD deployments through DevOps pipelines

Strong understanding of Medallion Architecture (Bronze, Silver, Gold layers) and associated data processing patterns

Experience building ingestion, transformation, and curation pipelines across Bronze, Silver, and Gold layers

Experience troubleshooting, monitoring, and supporting production data pipelines

Familiarity with Databricks Asset Bundles, Unity Catalog, and Delta Lake technologies is preferred

Healthcare industry experience preferred

Experience working with Medicare, payer, member enrollment, or claims data strongly preferred

Understanding of healthcare data quality, PHI/PII handling, and regulatory requirements preferred

Key Responsibilities

Design, develop, and deploy new Azure Databricks data pipelines using PySpark and Delta Live Tables

Translate and modernize legacy on-prem SQL pipelines into scalable cloud-native Databricks solutions

Build configuration-driven data pipelines utilizing YAML, JSON, and reusable framework patterns

Develop and maintain automated CI/CD deployment processes for data engineering workflows

Implement and support Medallion Architecture data pipelines across Bronze, Silver, and Gold layers

Partner with upstream and downstream teams to ensure data quality, reliability, and platform scalability

Troubleshoot and resolve production pipeline failures within both Databricks and legacy SQL environments

Enhance monitoring, observability, and operational support processes for healthcare data pipelines

Support Medicare and payer-focused data initiatives, including member, claims, and operational datasets

Follow established architecture patterns and engineering standards to ensure consistency across the platform