Databricks Engineer

Insight Global; United States; Remote; USD 140,000 - 190,000; Full time

Job summary

Insight Global is hiring a senior data engineer for a 100% remote US role. You will design and deploy Azure Databricks data pipelines using PySpark and Delta Live Tables, modernizing on-prem SQL pipelines into scalable cloud-native solutions.

You will build configuration-driven pipelines with YAML/JSON, maintain CI/CD for data workflows, and implement Medallion Architecture across Bronze, Silver, and Gold. Prior healthcare experience is a plus.

Qualifications

• 7+ years of data engineering experience building large-scale data platforms.
• Strong hands-on experience with Azure Databricks, Delta Lake, and ADLS.
• Advanced PySpark development experience, including DataFrames, transformations, joins, aggregations, and performance tuning.
• Experience developing and maintaining Delta Live Tables (DLT) pipelines.
• Strong SQL background with experience supporting legacy data warehouse environments.
• Experience migrating SQL-based pipelines to modern Python/PySpark architectures.
• Experience with Infrastructure as Code (IaC) and Pipeline as Code methodologies.
• Proficiency with YAML, JSON, and configuration-driven development patterns.
• Strong understanding of Medallion Architecture (Bronze, Silver, Gold).
• Experience building ingestion, transformation, and curation pipelines across Bronze, Silver, and Gold layers.
• Experience troubleshooting, monitoring, and supporting production data pipelines.
• Familiarity with Databricks Asset Bundles, Unity Catalog, and Delta Lake preferred.
• Healthcare industry experience preferred.
• Medicare, payer, member enrollment, or claims data experience strongly preferred.
• Understanding of healthcare data quality, PHI/PII handling, and regulatory requirements preferred.

Responsibilities

• Design, develop, and deploy Azure Databricks data pipelines using PySpark and Delta Live Tables.
• Modernize legacy on-prem SQL pipelines into scalable cloud-native Databricks solutions.
• Build configuration-driven data pipelines using YAML, JSON, and reusable framework patterns.
• Develop and maintain CI/CD deployment processes for data engineering workflows.
• Implement and support Medallion Architecture pipelines across Bronze, Silver, and Gold layers.
• Partner with upstream and downstream teams to ensure data quality, reliability, and scalability.
• Troubleshoot and resolve production pipeline issues across Databricks and legacy SQL environments.
• Enhance monitoring, observability, and operational support processes.
• Support Medicare and payer-focused initiatives involving member, claims, and operational datasets.
• Follow established architecture standards and engineering best practices to ensure platform consistency and maintainability.

Skills

PySpark

SQL

Data pipelines

IaC

YAML/JSON

Configuration-driven

Healthcare data concepts

Tools

Azure Databricks

Delta Lake

Delta Live Tables

ADLS

Unity Catalog

Databricks Asset Bundles

Job description

Location: 100% Remote in the US - CST/EST

Duration: 6-12 month contract + possible extensions or permanent conversion

Requirements

• 7+ years of Data Engineering experience building large-scale data platforms
• Strong hands-on experience with Azure Databricks, Delta Lake, and Azure Data Lake Storage (ADLS)
• Advanced PySpark development experience, including DataFrames, transformations, joins, aggregations, and performance tuning
• Experience developing and maintaining Delta Live Tables (DLT) pipelines
• Strong SQL background with experience supporting legacy data warehouse environments
• Experience migrating SQL-based pipelines to modern Python/PySpark architectures
• Experience with Infrastructure as Code (IaC) and Pipeline as Code methodologies
• Proficiency with YAML, JSON, and configuration-driven development patterns
• Strong understanding of Medallion Architecture (Bronze, Silver, Gold)
• Experience building ingestion, transformation, and curation pipelines across Bronze, Silver, and Gold layers
• Experience troubleshooting, monitoring, and supporting production data pipelines
• Familiarity with Databricks Asset Bundles, Unity Catalog, and Delta Lake preferred
• Healthcare industry experience preferred
• Medicare, payer, member enrollment, or claims data experience strongly preferred
• Understanding of healthcare data quality, PHI/PII handling, and regulatory requirements preferred

Key Responsibilities

• Design, develop, and deploy Azure Databricks data pipelines using PySpark and Delta Live Tables
• Modernize legacy on-prem SQL pipelines into scalable cloud-native Databricks solutions
• Build configuration-driven data pipelines using YAML, JSON, and reusable framework patterns
• Develop and maintain CI/CD deployment processes for data engineering workflows
• Implement and support Medallion Architecture pipelines across Bronze, Silver, and Gold layers
• Partner with upstream and downstream teams to ensure data quality, reliability, and scalability
• Troubleshoot and resolve production pipeline issues across Databricks and legacy SQL environments
• Enhance monitoring, observability, and operational support processes
• Support Medicare and payer-focused initiatives involving member, claims, and operational datasets
• Follow established architecture standards and engineering best practices to ensure platform consistency and maintainability