Azure Databricks Engineer - Scalable Data Lakes & Spark
IQuest Solutions Corporation
An application made for this job — a tailored resume and cover letter that speak straight to the posting.
Job summary
IQuest Solutions Corporation in Newtown Square, PA is seeking an experienced Azure Databricks Engineer to design, develop, and maintain scalable data solutions on Microsoft Azure. You will work with Databricks, Spark, Python, and SQL to build modern lakehouse architectures and reliable data pipelines.
Responsibilities include implementing Delta Lake, integrating with ADF and Synapse, and delivering CI/CD-enabled notebooks, jobs, and workflows.
Qualifications
- 10–15+ years of experience in data engineering or a related field.
- Strong hands-on experience with Azure Databricks.
- Strong knowledge of Apache Spark / PySpark.
- Proficiency in Python and SQL.
- Experience with Azure Data Lake Storage Gen2.
- Strong understanding of Delta Lake and lakehouse architecture.
- Experience with Azure Data Factory (ADF).
- Experience with Git and Azure DevOps CI/CD.
- Good understanding of data warehousing and ETL/ELT concepts.
- Experience troubleshooting and optimizing distributed data-processing workloads.
Responsibilities
- Design and develop scalable data pipelines using Azure Databricks, Apache Spark, Python, and SQL.
- Build and optimize ETL/ELT pipelines for batch and real-time data processing.
- Develop data solutions using ADLS Gen2 and Delta Lake.
- Implement Delta Lake tables, partitioning, schema evolution, and performance optimization.
- Develop notebooks, workflows, jobs, and reusable frameworks in Databricks.
- Integrate Databricks with Azure services (ADF, Key Vault, Synapse, Event Hubs).
- Perform data transformation, cleansing, validation, and quality checks.
- Optimize Spark jobs and Databricks workloads for performance and cost.
- Implement CI/CD pipelines using Azure DevOps and Git.
- Monitor production pipelines, troubleshoot failures, and resolve data-quality issues.
- Implement security, access control, and governance.
- Collaborate with data architects, analysts, developers, and business stakeholders.
Skills
Tools
Job description
IQuest Solutions Corporation in Newtown Square, PA is seeking an experienced Azure Databricks Engineer to design, develop, and maintain scalable data solutions on Microsoft Azure. You will work with Databricks, Spark, Python, and SQL to build modern lakehouse architectures and reliable data pipelines.
Responsibilities include implementing Delta Lake, integrating with ADF and Synapse, and delivering CI/CD-enabled notebooks, jobs, and workflows.
'/%3e%3cpath%20d='M101.744%2012.4726L102.894%2016.1947L99.0021%2013.2201L101.744%2012.4726ZM105.3%208.40937H100.488L99.0026%203.59473L97.5116%208.41047L92.7%208.40385L96.5965%2011.3834L95.1055%2016.1942L99.0021%2013.2196L101.408%2011.3834L105.3%208.40992V8.40937Z'%20fill='white'/%3e%3cdefs%3e%3clinearGradient%20id='paint0_linear_5593_1381'%20x1='90'%20y1='9.44471'%20x2='99'%20y2='9.44471'%20gradientUnits='userSpaceOnUse'%3e%3cstop%20stop-color='%2300AD70'/%3e%3cstop%20offset='0.998594'%20stop-color='%2300B67A'/%3e%3cstop%20offset='1'%20stop-color='%23DCDCE6'/%3e%3c/linearGradient%3e%3c/defs%3e%3c/svg%3e){kind=small}
