Azure Data Engineer — Lead Data Pipelines & Databricks

Quantum Technologies LLC

Pennsylvania

On-site

USD 91,000 - 124,000

Full time

24 hours ago
Be an early applicant

Get a reply from this recruiter

Your resume and cover letter, tailored to exactly what they’re hiring for.

Get past ATS filters

Job summary

Quantum Technologies. LLC in Malvern, PA is seeking an experienced Azure Data Engineer for a 12-month engagement to build and optimize large-scale data pipelines.

You will collect, store, and process data, develop ETL processes, and partner with business and tech teams to deliver reliable analytics solutions. You'll lead migration from Synapse to Databricks, implement batch and real-time pipelines with PySpark, Event Hubs, and Delta Lake, enforce data governance, and drive CI/CD automation using

Qualifications

  • 7+ years in data engineering, 3+ in a lead/senior role.
  • Strong expertise in Azure Databricks, Synapse, PySpark.
  • Proven experience with real-time streaming (Event Hubs, Kafka).
  • Deep knowledge of Azure ecosystem (Data Lake, Data Factory, Key Vault).
  • Strong SQL, performance tuning, and CI/CD pipeline skills.

Responsibilities

  • Collect, store, process and analyze large datasets to build ETL processes.
  • Lead migration of data pipelines from Synapse to Databricks.
  • Architect and implement batch and real-time data pipelines using PySpark and Event Hubs.
  • Establish data engineering best practices and mentor junior engineers.
  • Drive CI/CD automation with AzDO and Git.
  • Ensure data quality, governance, and compliance (Collibra).
  • Collaborate with business/tech teams and leverage insurance domain knowledge.
  • Develop reusable frameworks to reduce development effort and costs.
  • Develop quality code with performance optimizations at development stage.

Skills

Azure Databricks
PySpark
SQL
Data modeling
CI/CD
Leadership

Tools

Databricks
Event Hubs
Delta Lake
Kafka
Azure Data Factory
Cosmos DB

Job description

Quantum Technologies. LLC in Malvern, PA is seeking an experienced Azure Data Engineer for a 12-month engagement to build and optimize large-scale data pipelines.

You will collect, store, and process data, develop ETL processes, and partner with business and tech teams to deliver reliable analytics solutions. You'll lead migration from Synapse to Databricks, implement batch and real-time pipelines with PySpark, Event Hubs, and Delta Lake, enforce data governance, and drive CI/CD automation using

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Azure Data Engineer – ETL, Databricks & Data Lake Architect
Azure Data Engineer – ETL, Databricks & Data Lake Architect

Quantum Technologies. LLC • Atlanta (GA)

On-site
USD 120,000 - 150,000
Azure Data Engineer Lead - Databricks, Synapse, PySpark
Azure Data Engineer Lead - Databricks, Synapse, PySpark

Ethereum Technologies LLC • Pennsylvania

On-site
USD 89,000 - 126,000
Senior Azure Data Engineer: Pipelines & Databricks
Senior Azure Data Engineer: Pipelines & Databricks

Quantum Technologies LLC • Houston (TX)

On-site
USD 120,000 - 180,000
Azure Data Architect - Enterprise Data Strategy & Pipelines
Azure Data Architect - Enterprise Data Strategy & Pipelines

Quantum Technologies. LLC • Atlanta (GA)

On-site
USD 140,000 - 190,000
Azure Data Engineer — Pipelines, Databricks & Power BI
Azure Data Engineer — Pipelines, Databricks & Power BI

Quantum Technologies. LLC • Columbia Township (MO)

On-site
USD 90,000 - 130,000
Azure Data Engineer: Create Scalable Data Pipelines
Azure Data Engineer: Create Scalable Data Pipelines

Quantum Technologies LLC • Raleigh (NC)

On-site
USD 110,000 - 160,000
Azure Data Analytics Architect — Lead Databricks & Pipelines
Azure Data Analytics Architect — Lead Databricks & Pipelines

Quantum Technologies. LLC • Houston (TX)

Hybrid
USD 120,000 - 180,000
Data Engineer — Databricks & ADF Pipelines
Data Engineer — Databricks & ADF Pipelines

Quantum Technologies. LLC • Atlanta (GA)

On-site
USD 90,000 - 150,000
Azure Data Engineer Lead | Databricks & PySpark
Azure Data Engineer Lead | Databricks & PySpark

Robotics Technologies • Malvern

On-site
USD 89,000 - 126,000
Azure Data Engineer: ETL, Databricks & Synapse
Azure Data Engineer: ETL, Databricks & Synapse

Quantum Technologies. LLC • Austin (TX)

On-site
USD 120,000 - 180,000