Sr Azure Databricks Engineer

ITMC Systems, Inc; United States; Remote; USD 140,000 - 190,000; Full time

Job summary

ITMC Systems, Inc. is seeking an experienced Data Engineer to design and deploy Azure Databricks pipelines using PySpark and Delta Live Tables. You will modernize legacy SQL jobs into scalable cloud-native solutions, and build configuration-driven data workflows using YAML/JSON patterns.

Ideal candidates have 5–7+ years in data engineering, strong SQL, Python, and DevOps experience. Responsibilities include governance, end-to-end pipeline orchestration, and production support for

Qualifications

• 5-7+ years of Data Engineering experience building large-scale data platforms.
• Strong hands-on experience with Azure Databricks, Delta Lake, and Azure Data Lake Storage (ADLS).
• Advanced PySpark development experience, including DataFrames, transformations, joins, aggregations, and performance tuning.
• Experience developing and maintaining Delta Live Tables (DLT) pipelines.
• Strong SQL background with experience supporting legacy data warehouse environments.
• Experience migrating SQL-based pipelines to modern Python/PySpark architectures.
• Experience with Infrastructure as Code (IaC) and Pipeline as Code methodologies.
• Proficiency with YAML, JSON, and configuration-driven development patterns.
• Experience automating CI/CD deployments using DevOps pipelines.
• Strong understanding of Medallion Architecture (Bronze, Silver, Gold).
• Experience building ingestion, transformation, and curation pipelines across Bronze, Silver, and Gold layers.
• Experience troubleshooting, monitoring, and supporting production data pipelines.

Responsibilities

• Design, develop, and deploy Azure Databricks data pipelines using PySpark and Delta Live Tables.
• Modernize legacy on-prem SQL pipelines into scalable cloud-native Databricks solutions.
• Build configuration-driven data pipelines using YAML, JSON, and reusable framework patterns.
• Develop and maintain CI/CD deployment processes for data engineering workflows.
• Implement and support Medallion Architecture pipelines across Bronze, Silver, and Gold layers.
• Partner with upstream and downstream teams to ensure data quality, reliability, and scalability.
• Troubleshoot and resolve production pipeline issues across Databricks and legacy SQL environments.
• Enhance monitoring, observability, and operational support processes.
• Support Medicare and payer-focused initiatives involving member, claims, and operational datasets.
• Follow established architecture standards and engineering best practices to ensure platform consistency and maintainability.

Skills

PySpark development

SQL

Python

Data pipelines design

CI/CD/DevOps

Data quality frameworks

Delta Live Tables

Medallion Architecture

Production troubleshooting

Tools

Azure Databricks

Delta Lake

Azure Data Lake Storage (ADLS)

Job description

Core requirement: Strong data engineering fundamentals with proven, hands-on Databricks experience. Deep experience of Databricks concepts the team uses day to day.

Technical must-haves: Performance tuning and optimization, including Spark internals, join strategies, and data skew handling; strong use of LDP APIs, CDF, and CDC; and experience building data-quality frameworks using expectations, quarantine logic, and alerting.

Governance & orchestration: Hands-on experience with PII/PHI masking, access controls, compliance/governance, plus the ability to orchestrate and stitch Databricks pipelines together end to end.

Location: Remote- US

Required Skills & Experience

• 5-7+ years of Data Engineering experience building large-scale data platforms
• Strong hands-on experience with Azure Databricks, Delta Lake, and Azure Data Lake Storage (ADLS)
• Advanced PySpark development experience, including DataFrames, transformations, joins, aggregations, and performance tuning
• Experience developing and maintaining Delta Live Tables (DLT) pipelines
• Strong SQL background with experience supporting legacy data warehouse environments
• Experience migrating SQL-based pipelines to modern Python/PySpark architectures
• Experience with Infrastructure as Code (IaC) and Pipeline as Code methodologies
• Proficiency with YAML, JSON, and configuration-driven development patterns
• Experience automating CI/CD deployments using DevOps pipelines
• Strong understanding of Medallion Architecture (Bronze, Silver, Gold)
• Experience building ingestion, transformation, and curation pipelines across Bronze, Silver, and Gold layers
• Experience troubleshooting, monitoring, and supporting production data pipelines
• Familiarity with Databricks Asset Bundles, Unity Catalog, and Delta Lake preferred
• Healthcare industry experience preferred
• Medicare, payer, member enrollment, or claims data experience strongly preferred
• Understanding of healthcare data quality, PHI/PII handling, and regulatory requirements preferred

Job Description

• Design, develop, and deploy Azure Databricks data pipelines using PySpark and Delta Live Tables
• Modernize legacy on-prem SQL pipelines into scalable cloud-native Databricks solutions
• Build configuration-driven data pipelines using YAML, JSON, and reusable framework patterns
• Develop and maintain CI/CD deployment processes for data engineering workflows
• Implement and support Medallion Architecture pipelines across Bronze, Silver, and Gold layers
• Partner with upstream and downstream teams to ensure data quality, reliability, and scalability
• Troubleshoot and resolve production pipeline issues across Databricks and legacy SQL environments
• Enhance monitoring, observability, and operational support processes
• Support Medicare and payer-focused initiatives involving member, claims, and operational datasets
• Follow established architecture standards and engineering best practices to ensure platform consistency and maintainability