Sr. Azure Data Engineer with Databricks

VeeRteq Solutions Inc.; Philadelphia (Philadelphia County); Hybrid; USD 120,000 - 160,000; Full time

Benefits offered by this job

Hybrid work model

Onsite presence three days per week

Job summary

VeeRteq Solutions Inc. in Philadelphia is seeking a senior Data Engineer to build and optimize enterprise Azure and Databricks data platforms in a hybrid role.

You will design scalable ETL/ELT pipelines, ingest, transform, and orchestrate data using Azure-native tools, Databricks, PySpark, and Delta Lake, while ensuring data quality and governance. The role requires onsite presence three days per week and collaboration with business stakeholders.

Qualifications

• Bachelor’s degree in Computer Science or a related field.
• 7+ years in Data Engineering.
• 5+ years hands-on experience with Azure data technologies.
• 2+ years experience in Healthcare, Managed Care, Health Plans, or Payer organizations.
• Strong expertise in Azure Databricks, ADLS Gen2, ADF, PySpark, Python, SQL, Delta Lake, and Data Modeling.
• Experience building enterprise-scale data pipelines and data integration.
• Familiarity with Data Warehousing, Data Lake, and Lakehouse concepts.
• Experience with Git, CI/CD, and DevOps practices.

Responsibilities

• Design, develop, and maintain scalable ETL/ELT pipelines for enterprise data and analytics initiatives
• Build data ingestion, transformation, and orchestration frameworks using Azure-native technologies
• Develop and optimize data processing solutions using Databricks and PySpark
• Implement and maintain Delta Lake and Lakehouse architectures
• Ensure data quality, accuracy, lineage, and reliability across data assets
• Perform root cause analysis and troubleshoot data-related issues
• Collaborate with Product Owners, Architects, QA teams, and business stakeholders
• Implement CI/CD processes for data engineering solutions and support automated deployment and release management
• Develop monitoring, logging, and observability capabilities for data pipelines
• Follow enterprise data governance and security standards
• Participate in Agile/Scrum ceremonies including sprint planning, daily stand-ups, retrospectives, and backlog refinement
• Contribute to continuous improvement and engineering best practices

Skills

Azure Databricks

Azure Data Lake

Azure Data Factory

PySpark

Python

SQL

Delta Lake

Data Modeling

ETL/ELT Dev

Azure Cloud Platform

Git

CI/CD

Agile/Scrum

Azure Synapse Analytics

Azure SQL Database

Education

Bachelor’s degree in Computer Science, Information Systems, Engineering, or related field

Tools

Azure Databricks

Azure Data Lake Gen2

Azure Data Factory

Databricks

Azure Synapse Analytics

Power BI

Job description

Build and optimize enterprise Azure and Databricks data platforms in a hybrid role based in Philadelphia.

Responsibilities

• Design, develop, and maintain scalable ETL/ELT pipelines for enterprise data and analytics initiatives
• Build data ingestion, transformation, and orchestration frameworks using Azure-native technologies
• Develop and optimize data processing solutions using Databricks and PySpark
• Implement and maintain Delta Lake and Lakehouse architectures
• Ensure data quality, accuracy, lineage, and reliability across data assets
• Perform root cause analysis and troubleshoot data-related issues
• Work on solutions leveraging:
  • Azure Databricks
  • Azure Data Lake Storage (ADLS Gen2)
  • Azure Data Factory (ADF)
  • Azure Synapse Analytics
  • Azure SQL Database
• Support data migration, modernization, and cloud transformation efforts
• Optimize performance, scalability, and cost efficiency for cloud-based data platforms
• Build and maintain logical and physical data models
• Create curated data layers for BI, reporting, and advanced analytics
• Partner with business users and data consumers to translate requirements into scalable technical solutions
• Implement CI/CD processes for data engineering solutions and support automated deployment and release management
• Develop monitoring, logging, and observability capabilities for data pipelines
• Follow enterprise data governance and security standards
• Participate in Agile/Scrum ceremonies including sprint planning, daily stand-ups, retrospectives, and backlog refinement
• Collaborate with Product Owners, Architects, QA teams, and business stakeholders
• Contribute to continuous improvement and engineering best practices

Requirements

• Bachelor’s degree in Computer Science, Information Systems, Engineering, or a related field
• 7+ years of experience in Data Engineering
• 5+ years hands-on experience with Azure data technologies
• 2+ years experience in Healthcare, Managed Care, Health Plans, or Payer organizations
• Strong expertise in:
  • Azure Databricks
  • Azure Data Lake Storage (ADLS)
  • Azure Data Factory (ADF)
  • PySpark
  • Python
  • SQL
  • Delta Lake and Data Modeling
• Extensive experience building enterprise-scale data pipelines and data integration solutions
• Strong understanding of Data Warehousing, Data Lake, and Lakehouse concepts
• Experience with large volumes of structured and unstructured data
• Experience with Git, CI/CD, and DevOps practices
• Strong analytical, troubleshooting, and problem‑solving skills

Required Technical Skills

• Must Have: Azure Databricks
• Must Have: Azure Data Lake (ADLS Gen2)
• Must Have: Azure Data Factory (ADF)
• Must Have: PySpark
• Must Have: Python
• Must Have: SQL
• Must Have: Delta Lake
• Must Have: Data Modeling
• Must Have: ETL/ELT Development
• Must Have: Azure Cloud Platform
• Must Have: Git
• Must Have: CI/CD
• Must Have: Agile/Scrum
• Skills: Azure Synapse Analytics, Azure SQL Database

Preferred Qualifications

• Experience working with: Claims Data, Membership Data, Provider Data, Care Management Data
• Experience with Power BI or other reporting tools
• Familiarity with FHIR, HL7, or healthcare interoperability standards
• Experience with Azure DevOps and Infrastructure as Code (IaC)
• Exposure to AI/ML data pipelines and advanced analytics environments

Location & Work Model

• Philadelphia, PA (hybrid)
• Onsite presence required at least three days per week

Technology Stack

• Azure, Databricks, Azure Databricks, PySpark, Python, SQL
• Delta Lake and Lakehouse architectures, ETL/ELT
• Azure Data Lake Storage (ADLS Gen2), Azure Data Lake Storage (ADLS)
• Azure Data Factory (ADF), Azure Synapse Analytics, Azure SQL Database
• Git, CI/CD, Agile/Scrum, Data Modeling