Senior Azure Databricks Engineer — Lakehouse Lead

Global Technical Talent, an Inc. 5000 Company; Glendale (AZ); Hybrid; USD 107,000 - 139,000; Full time

Benefits offered by this job

Medical, Vision, and Dental Insurance

401k Retirement Fund

Job summary

This Senior Azure Databricks Engineer role is based in Glendale, AZ and requires 8 to 10 years of experience. The engineer will collaborate with the Data Platforms and Integration team to architect and implement scalable enterprise data platform solutions using Azure Databricks, Delta Lake, Unity Catalog, and related Azure services.

The position emphasizes designing data engineering frameworks for batch, streaming, ELT/ETL, and real-time analytics, along with governance, security, and

Qualifications

• At least 8-10 years of experience in designing and implementing enterprise data platforms.
• At least 5 years of hands-on Azure Databricks experience in enterprise environments.
• Experience with architecting modern Lakehouse solutions using Azure Databricks and Delta Lake.
• Experience administering Azure Databricks workspaces, clusters, Unity Catalog, compute policies, job orchestration, and workspace security.
• Experience developing data engineering solutions using Apache Spark (PySpark and/or Spark SQL).

Responsibilities

• Collaborate with Data Platforms and Integration team in architecting and implementing scalable enterprise data platform solutions utilizing Azure Databricks, Delta Lake, Unity Catalog, Azure Data Lake Storage Gen2, and related Azure services.
• Design and develop enterprise data engineering frameworks supporting batch, streaming, ELT/ETL, and real-time analytics workloads.
• Administer and maintain Azure Databricks workspaces including cluster configuration, job scheduling, workspace administration, Unity Catalog implementation, secret management, compute policies, cluster policies, and workspace governance.
• Define and implement enterprise security architecture including identity integration with Microsoft Entra ID (Azure AD), RBAC, Privileged Identity Management (PIM), data encryption, private networking, managed identities, Key Vault integration, and secure access patterns.
• Establish enterprise data governance standards utilizing Unity Catalog including data lineage, data quality, metadata management, catalog organization, access policies, and audit capabilities.
• Optimize Databricks workloads through performance tuning, Spark optimization, cluster sizing, workload isolation, caching strategies, partitioning, Delta optimization, Photon, and cost optimization best practices.
• Design enterprise monitoring, logging, operational dashboards, alerting, and platform health monitoring utilizing Azure Monitor, Log Analytics, Application Insights, and Databricks system tables.
• Develop reusable architecture patterns, reference implementations, development standards, naming conventions, coding standards, operational procedures, and platform documentation.
• Support migration of legacy enterprise data warehouses, ETL solutions, and reporting platforms to Azure Databricks and modern Lakehouse architecture.
• Provide technical support and troubleshooting for production incidents related to Azure Databricks, Spark workloads, data pipelines, and platform infrastructure.

Skills

Azure Databricks

Delta Lake

Unity Catalog

Azure Data Lake Gen2

Azure services

Apache Spark

PySpark

Spark SQL

RBAC

Key Vault

Azure DevOps

Data governance

Tools

Azure Databricks

Delta Lake

Job description

This Senior Azure Databricks Engineer role is based in Glendale, AZ and requires 8 to 10 years of experience. The engineer will collaborate with the Data Platforms and Integration team to architect and implement scalable enterprise data platform solutions using Azure Databricks, Delta Lake, Unity Catalog, and related Azure services.

The position emphasizes designing data engineering frameworks for batch, streaming, ELT/ETL, and real-time analytics, along with governance, security, and