Sr Data Engineer (Azure Databricks)

Insight Global (United States)

Woonsocket (RI)

On-site

USD 120,000 - 160,000

Full time

13 days ago

Get a reply from this employer

Your resume and cover letter, tailored to exactly what they’re hiring for.

Get past ATS filters

Job summary

Insight Global in the United States is seeking a Data Engineer to design, develop, and deploy Azure Databricks data pipelines using PySpark and Delta Live Tables. The role focuses on modernizing legacy on-prem pipelines into scalable cloud-native Databricks solutions and building configuration-driven pipelines with YAML and JSON.

You will collaborate with upstream and downstream teams to ensure data quality and reliability, implement CI/CD for data workflows, and support Medicare and payer

Qualifications

  • 5-7+ years of Data Engineering experience building large-scale data platforms.
  • Strong hands-on experience with Azure Databricks, Delta Lake, and Azure Data Lake Storage (ADLS).
  • Advanced PySpark development experience, including DataFrames, transformations, joins, aggregations, and performance tuning.
  • Experience developing and maintaining Delta Live Tables (DLT) pipelines.
  • Strong SQL background with experience supporting legacy data warehouse environments.

Responsibilities

  • Design, develop, and deploy Azure Databricks data pipelines using PySpark and Delta Live Tables.
  • Modernize legacy on-prem SQL pipelines into scalable cloud-native Databricks solutions.
  • Build configuration-driven data pipelines using YAML, JSON, and reusable framework patterns.
  • Develop and maintain CI/CD deployment processes for data engineering workflows.
  • Implement and support Medallion Architecture pipelines across Bronze,Silver, and Gold layers.

Skills

PySpark
Azure Databricks
Delta Lake
SQL
IaC
CI/CD
YAML/JSON
Medallion Architecture
Delta Live Tables
Data pipelines

Tools

Azure Databricks
Delta Live Tables
Unity Catalog

Job description

Job Description

  • Design, develop, and deploy Azure Databricks data pipelines using PySpark and Delta Live Tables
  • Modernize legacy on-prem SQL pipelines into scalable cloud-native Databricks solutions
  • Build configuration-driven data pipelines using YAML, JSON, and reusable framework patterns
  • Develop and maintain CI/CD deployment processes for data engineering workflows
  • Implement and support Medallion Architecture pipelines across Bronze, Silver, and Gold layers
  • Partner with upstream and downstream teams to ensure data quality, reliability, and scalability
  • Troubleshoot and resolve production pipeline issues across Databricks and legacy SQL environments
  • Enhance monitoring, observability, and operational support processes
  • Support Medicare and payer-focused initiatives involving member, claims, and operational datasets
  • Follow established architecture standards and engineering best practices to ensure platform consistency and maintainability

Skills and Requirements

  • 5-7+ years of Data Engineering experience building large-scale data platforms
  • Strong hands-on experience with Azure Databricks, Delta Lake, and Azure Data Lake Storage (ADLS)
  • Advanced PySpark development experience, including DataFrames, transformations, joins, aggregations, and performance tuning
  • Experience developing and maintaining Delta Live Tables (DLT) pipelines
  • Strong SQL background with experience supporting legacy data warehouse environments
  • Experience migrating SQL-based pipelines to modern Python/PySpark architectures
  • Experience with Infrastructure as Code (IaC) and Pipeline as Code methodologies
  • Proficiency with YAML, JSON, and configuration-driven development patterns
  • Experience automating CI/CD deployments using DevOps pipelines
  • Strong understanding of Medallion Architecture (Bronze, Silver, Gold)
  • Experience building ingestion, transformation, and curation pipelines across Bronze, Silver, and Gold layers
  • Experience troubleshooting, monitoring, and supporting production data pipelines
  • Familiarity with Databricks Asset Bundles, Unity Catalog, and Delta Lake preferred
  • Healthcare industry experience preferred
  • Medicare, payer, member enrollment, or claims data experience strongly preferred
  • Understanding of healthcare data quality, PHI/PII handling, and regulatory requirements preferred

Important Core Databricks concepts we rely on daily: We need candidates with solid data engineering fundamentals and proven hands-on Databricks experience, specifically across:

  • Performance tuning & optimization (Spark internals, join strategies, skew handling)
  • Efficient use of LDP APIs (CDF, CDC)
  • Data quality frameworks (expectations, quarantine logic, alerting)
  • Compliance & governance (PII/PHI masking, access controls)
  • and lastly on Orchestrating and stiching these pipelines together

We are a company committed to creating diverse and inclusive environments where people can bring their full, authentic selves to work every day. We are an equal employment opportunity/affirmative action employer that believes everyone matters. Qualified candidates will receive consideration for employment without regard to race, color, ethnicity, religion, sex (including pregnancy), sexual orientation, gender identity and expression, marital status, national origin, ancestry, genetic factors, age, disability, protected veteran status, military or uniformed service member status, or any other status or characteristic protected by applicable laws, regulations, and ordinances. If you need assistance and/or a reasonable accommodation due to a disability during the application or the recruiting process, please send a request to HR@insightglobal.com.

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Sr Azure Databricks Engineer
Sr Azure Databricks Engineer

ITMC Systems, Inc • United States

Remote
USD 140,000 - 190,000
Databricks Engineer
Databricks Engineer

Insight Global (United States) • United States

Remote
USD 140,000 - 190,000
Databricks Data Engineer
Databricks Data Engineer

Quantum Technologies LLC • Dallas (TX)

On-site
USD 150,000 - 190,000
Senior Data Engineer on-site)
Senior Data Engineer on-site)

Ziosk • Dallas (TX)

On-site
USD 140,000 - 190,000
Databricks Engineer
Databricks Engineer

Princeton IT Services, Inc • United States

Remote
USD 130,000 - 180,000
Databricks Data Engineer
Databricks Data Engineer

Robotic Process Automation LLC • Dallas (TX)

On-site
USD 140,000 - 180,000
Databricks Data Engineer
Databricks Data Engineer

Valiant Technologies LLC • Dallas (TX)

On-site
USD 120,000 - 190,000
Databricks Data Engineer
Databricks Data Engineer

Ethereum Technologies LLC • Dallas (TX)

On-site
USD 120,000 - 170,000
Databricks Engineer
Databricks Engineer

Robotics Technologies LLC • Austin (TX)

On-site
USD 130,000 - 170,000
Databricks Engineer
Databricks Engineer

Ethereum Technologies LLC • Austin (TX)

On-site
USD 130,000 - 185,000