Senior Azure Databricks Engineer

Airswift; Glendale (AZ); On-site; USD 140,000 - 170,000; Full time

Job summary

Airswift is seeking a Senior Azure Databricks Engineer in Glendale, AZ to design and implement scalable enterprise data platforms in a modern Azure environment. You will work across batch, streaming, ELT/ETL, and real-time analytics workloads with emphasis on Databricks, Delta Lake, Unity Catalog, and ADLS Gen2.

This role involves platform architecture, governance, security, performance optimisation, and migration towards a Lakehouse architecture while supporting enterprise data pipelines and

Qualifications

• 8–10 years of experience designing and implementing enterprise data platforms.
• 5+ years of hands-on experience with Azure Databricks in enterprise environments.
• Experience architecting modern Lakehouse solutions using Azure Databricks and Delta Lake.
• Experience administering Azure Databricks workspaces, clusters, Unity Catalog, compute policies, job orchestration, and workspace security.
• Experience implementing enterprise data governance including metadata management, data lineage, and fine-granular access controls.
• Experience designing enterprise security architectures including RBAC, managed identities, private endpoints, Key Vault integration, encryption, and network security.
• Experience implementing enterprise data quality frameworks, monitoring, observability, and operational dashboards.
• Experience supporting migration of legacy enterprise data warehouses, ETL solutions, and reporting platforms to Azure Databricks and modern Lakehouse architecture.
• Experience providing technical support and troubleshooting for production incidents related to Databricks, Spark workloads, data pipelines, and platform infrastructure

Skills

Azure Databricks

Delta Lake

Unity Catalog

Azure Data Lake Gen2

Apache Spark

PySpark

Spark SQL

SQL

Python

Data Modeling

Security Architecture

CI/CD Azure DevOps

Monitoring & Dashboards

Cost Optimization

Tools

Azure DevOps

Azure Monitor

Log Analytics

Application Insights

Databricks

Job description

Job Title: Senior Azure Databricks Engineer Duration: 12 months Schedule: 5-40 schedule Location: Glendale, AZ

This opportunity is focused on designing and implementing scalable enterprise data platform solutions in a modern Azure environment. The role will support enterprise data engineering frameworks across batch, streaming, ELT/ETL, and real-time analytics workloads, with strong emphasis on Azure Databricks, Delta Lake, Unity Catalog, and Azure Data Lake Storage Gen2. This position offers involvement in platform architecture, governance, security, performance optimisation, and migration to a modern Lakehouse architecture.

Required Experience

• At least 8-10 years of experience designing and implementing enterprise data platforms


• At least 5 years of hands‑on experience with Azure Databricks in enterprise environments


• Experience architecting modern Lakehouse solutions using Azure Databricks and Delta Lake


• Experience administering Azure Databricks workspaces, clusters, Unity Catalog, compute policies, job orchestration, and workspace security


• Experience implementing enterprise data governance including metadata management, data lineage, and fine‑granular access controls


• Experience designing enterprise security architectures including RBAC, managed identities, private endpoints, Key Vault integration, encryption, and network security


• Experience implementing enterprise data quality frameworks, monitoring, observability, and operational dashboards


• Experience supporting migration of legacy enterprise data warehouses, ETL solutions, and reporting platforms to Azure Databricks and modern Lakehouse architecture


• Experience providing technical support and troubleshooting for production incidents related to Databricks, Spark workloads, data pipelines, and platform infrastructure

Required Skills

• Strong capability in designing and developing enterprise data engineering frameworks for batch, streaming, ELT/ETL, and real-time analytics workloads


• Advanced knowledge of Azure Databricks, Delta Lake, Unity Catalog, and Azure Data Lake Storage Gen2


• Strong experience with Apache Spark, including PySpark and Spark SQL development


• Strong SQL programming skills with proficiency in Python and PySpark


• Strong understanding of data modelling techniques including dimensional modelling, Data Vault, medallion architecture, and enterprise data warehouse concepts


• Knowledge of Azure Key Vault, Azure Monitor, Microsoft Entra ID, and Azure networking


• Ability to optimise Databricks workloads through Spark tuning, cluster sizing, workload isolation, caching, partitioning, Delta optimisation, Photon, and cost optimisation practices


• Ability to design enterprise monitoring, logging, dashboards, alerting, and platform health monitoring using Azure Monitor, Log Analytics, Application Insights, and Databricks system tables


• Experience implementing CI/CD pipelines using Azure DevOps, automated testing, release management, and deployment automation


• Ability to develop reusable architecture patterns, reference implementations, development standards, naming conventions, coding standards, operational procedures, and platform documentation

About Airswift

Airswift is an international workforce solutions provider within the energy, process and infrastructure industries. Airswift serves as a strategic partner to clients, offering a turnkey workforce solution to capture and deliver the top talent needed to complete successful projects by aligning with unique project needs. With over 1000 employees and 9,000 contractors operating in over 60 countries, Airswift's geographical reach and pool of talent available is unmatched in the industry.