Azure Databricks Engineer

IQuest Solutions Corporation; Newtown Square (Delaware County); On-site; USD 120,000 - 170,000; Full time

Job summary

IQuest Solutions Corporation in Newtown Square, PA is seeking an experienced Azure Databricks Engineer to design, develop, and maintain scalable data solutions on Microsoft Azure. You will work with Databricks, Spark, Python, and SQL to build modern lakehouse architectures and reliable data pipelines.

Responsibilities include implementing Delta Lake, integrating with ADF and Synapse, and delivering CI/CD-enabled notebooks, jobs, and workflows.

Qualifications

• 10–15+ years of experience in data engineering or a related field.
• Strong hands-on experience with Azure Databricks.
• Strong knowledge of Apache Spark / PySpark.
• Proficiency in Python and SQL.
• Experience with Azure Data Lake Storage Gen2.
• Strong understanding of Delta Lake and lakehouse architecture.
• Experience with Azure Data Factory (ADF).
• Experience with Git and Azure DevOps CI/CD.
• Good understanding of data warehousing and ETL/ELT concepts.
• Experience troubleshooting and optimizing distributed data-processing workloads.

Responsibilities

• Design and develop scalable data pipelines using Azure Databricks, Apache Spark, Python, and SQL.
• Build and optimize ETL/ELT pipelines for batch and real-time data processing.
• Develop data solutions using ADLS Gen2 and Delta Lake.
• Implement Delta Lake tables, partitioning, schema evolution, and performance optimization.
• Develop notebooks, workflows, jobs, and reusable frameworks in Databricks.
• Integrate Databricks with Azure services (ADF, Key Vault, Synapse, Event Hubs).
• Perform data transformation, cleansing, validation, and quality checks.
• Optimize Spark jobs and Databricks workloads for performance and cost.
• Implement CI/CD pipelines using Azure DevOps and Git.
• Monitor production pipelines, troubleshoot failures, and resolve data-quality issues.
• Implement security, access control, and governance.
• Collaborate with data architects, analysts, developers, and business stakeholders.

Skills

Data engineering

Cloud data platforms

ETL/ELT design

Problem solving

Tools

Azure Databricks

Apache Spark / PySpark

Python

SQL

Azure Data Lake Storage Gen2

Delta Lake

Azure Data Factory

Git / Azure DevOps CI/CD

Power BI

Azure Synapse Analytics

Job description

Job SummaryWe are looking for an experienced

Location: Newtown Sqaure, PA

W2 POSITION

Job Summary

We are looking for an experienced Azure Databricks Engineer to design, develop, and maintain scalable data engineering solutions on Microsoft Azure. The ideal candidate will have strong hands‑on experience with Azure Databricks, Apache Spark, Python/SQL, and modern data lake architectures.

Key Responsibilities

• Design and develop scalable data pipelines using Azure Databricks, Apache Spark, Python, and SQL.
• Build and optimize ETL/ELT pipelines for batch and real‑time data processing.
• Develop data solutions using Azure Data Lake Storage (ADLS Gen2) and Delta Lake.
• Implement Delta Lake tables, partitioning, schema evolution, and performance optimization.
• Develop notebooks, workflows, jobs, and reusable frameworks in Azure Databricks.
• Integrate Databricks with Azure services such as Azure Data Factory, Azure Key Vault, Azure Synapse Analytics, and Azure Event Hubs.
• Perform data transformation, cleansing, validation, and quality checks.
• Optimize Spark jobs and Databricks workloads for performance and cost.
• Implement CI/CD pipelines using Azure DevOps and Git.
• Monitor production pipelines, troubleshoot failures, and resolve data‑quality issues.
• Implement security, access control, and governance following enterprise standards.
• Collaborate with data architects, analysts, developers, and business stakeholders.

Required Skills

• 10 15+ years of experience in data engineering or a related field.
• Strong hands‑on experience with Azure Databricks.
• Strong knowledge of Apache Spark / PySpark.
• Proficiency in Python and SQL.
• Experience with Azure Data Lake Storage Gen2.
• Strong understanding of Delta Lake and lakehouse architecture.
• Experience with Azure Data Factory (ADF).
• Experience with Git and Azure DevOps CI/CD.
• Good understanding of data warehousing and ETL/ELT concepts.
• Experience troubleshooting and optimizing distributed data‑processing workloads.

Preferred Skills

• Experience with Azure Synapse Analytics.
• Knowledge of Unity Catalog and Databricks governance.
• Experience with streaming technologies such as Event Hubs, Kafka, or Structured Streaming.
• Experience with Terraform or infrastructure-as-code.
• Knowledge of data modeling and dimensional modeling.
• Experience implementing data quality and observability frameworks.
• Familiarity with Power BI and downstream analytics platforms.