Sr Data Engineer (Azure Databricks)

Insight Global; Woonsocket (RI); On-site; USD 120,000 - 160,000; Full time

Job summary

Insight Global is seeking a senior Data Engineer in the United States to design, build, and deploy Azure Databricks pipelines using PySpark and Delta Live Tables. You will modernize legacy SQL pipelines into scalable cloud-native Databricks solutions and develop configuration-driven data pipelines with YAML/JSON patterns.

You will also implement CI/CD for data workflows, support Medallion Architecture across Bronze/Silver/Gold, and ensure data quality with upstream/downstream partners.

Qualifications

• 5-7+ years of Data Engineering experience building large-scale data platforms.
• Strong hands-on experience with Azure Databricks, Delta Lake, and ADLS.
• Advanced PySpark development and SQL knowledge, incl. performance tuning.

Responsibilities

• Design, develop, and deploy Azure Databricks data pipelines using PySpark and Delta Live Tables.
• Modernize legacy on-prem SQL pipelines into scalable cloud-native Databricks solutions.
• Build configuration-driven data pipelines using YAML, JSON, and reusable framework patterns.
• Develop and maintain CI/CD deployment processes for data engineering workflows.
• Implement and support Medallion Architecture pipelines across Bronze, Silver, and Gold layers.
• Partner with upstream and downstream teams to ensure data quality, reliability, and scalability.
• Troubleshoot and resolve production pipeline issues across Databricks and legacy SQL environments.
• Enhance monitoring, observability, and operational support processes.
• Support Medicare and payer-focused initiatives involving member, claims, and operational datasets.
• Follow established architecture standards and engineering best practices to ensure platform consistency and maintainability.

Skills

Data Engineering

Python / PySpark

SQL

Cloud data platforms

Tools

Azure Databricks

Delta Lake

Azure Data Lake Storage

Delta Live Tables

Unity Catalog

Databricks Asset Bundles

IaC (Terraform/ARM)

YAML/JSON

Job description

Job Description

• Design, develop, and deploy Azure Databricks data pipelines using PySpark and Delta Live Tables
• Modernize legacy on-prem SQL pipelines into scalable cloud-native Databricks solutions
• Build configuration-driven data pipelines using YAML, JSON, and reusable framework patterns
• Develop and maintain CI/CD deployment processes for data engineering workflows
• Implement and support Medallion Architecture pipelines across Bronze, Silver, and Gold layers
• Partner with upstream and downstream teams to ensure data quality, reliability, and scalability
• Troubleshoot and resolve production pipeline issues across Databricks and legacy SQL environments
• Enhance monitoring, observability, and operational support processes
• Support Medicare and payer-focused initiatives involving member, claims, and operational datasets
• Follow established architecture standards and engineering best practices to ensure platform consistency and maintainability

Skills and Requirements

• 5-7+ years of Data Engineering experience building large-scale data platforms
• Strong hands-on experience with Azure Databricks, Delta Lake, and Azure Data Lake Storage (ADLS)
• Advanced PySpark development experience, including DataFrames, transformations, joins, aggregations, and performance tuning
• Experience developing and maintaining Delta Live Tables (DLT) pipelines
• Strong SQL background with experience supporting legacy data warehouse environments
• Experience migrating SQL-based pipelines to modern Python/PySpark architectures
• Experience with Infrastructure as Code (IaC) and Pipeline as Code methodologies
• Proficiency with YAML, JSON, and configuration-driven development patterns
• Experience automating CI/CD deployments using DevOps pipelines
• Strong understanding of Medallion Architecture (Bronze, Silver, Gold)
• Experience building ingestion, transformation, and curation pipelines across Bronze, Silver, and Gold layers
• Experience troubleshooting, monitoring, and supporting production data pipelines
• Familiarity with Databricks Asset Bundles, Unity Catalog, and Delta Lake preferred
• Healthcare industry experience preferred
• Medicare, payer, member enrollment, or claims data experience strongly preferred
• Understanding of healthcare data quality, PHI/PII handling, and regulatory requirements preferred

We are a company committed to creating diverse and inclusive environments where people can bring their full, authentic selves to work every day. We are an equal employment opportunity/affirmative action employer that believes everyone matters. Qualified candidates will receive consideration for employment without regard to race, color, ethnicity, religion, sex (including pregnancy), sexual orientation, gender identity and expression, marital status, national origin, ancestry, genetic factors, age, disability, protected veteran status, military or uniformed service member status, or any other status or characteristic protected by applicable laws, regulations, and ordinances. If you need assistance and/or a reasonable accommodation due to a disability during the application or the recruiting process, please send a request to [e-mail].