Senior Azure Databricks Engineer

Environmental and Safety Solutions, Inc.; Glendale (AZ); On-site; USD 140,000 - 180,000; Full time

Job summary

Environmental and Safety Solutions, Inc. is seeking a Senior Azure Databricks Engineer to design, implement, and govern enterprise data platforms on Azure.

You will work with cross-functional teams to build scalable data pipelines, enforce security controls, and drive the Lakehouse initiative using Databricks, Unity Catalog, and Delta Lake. Responsibilities include workspace administration, governance, performance optimization, monitoring, and migration of legacy systems to modern Data Lakehouse

Qualifications

• 8–10 years designing enterprise data platforms.
• 5+ years hands-on Azure Databricks in enterprise environments.
• Experience with Delta Lake and Unity Catalog in Lakehouse architecture.
• Strong SQL and Python/PySpark development skills.
• Experience with security, governance, and RBAC in Azure.
• Experience with CI/CD pipelines using Azure DevOps.

Responsibilities

• Collaborate with Data Platforms and Integration teams to architect scalable data platform solutions on Azure Databricks.
• Design and develop enterprise data engineering frameworks for batch, streaming, ELT/ETL, and real-time analytics.
• Administer and maintain Azure Databricks workspaces, Unity Catalog, and security configurations.
• Define and implement enterprise security architecture with Entra ID, RBAC, PIM, Key Vault.
• Establish data governance standards using Unity Catalog including lineage and metadata management.
• Optimize workloads with Spark tuning, Delta optimization, and cost controls.
• Develop monitoring, dashboards, and platform health using Azure Monitor and Log Analytics.
• Create reusable architecture patterns and documentation.
• Support migration of legacy data warehouses to Lakehouse architecture.
• Provide incident response and troubleshooting for Databricks and data pipelines.

Skills

Python

PySpark

SQL

Data modeling

CI/CD

Tools

Azure Databricks

Delta Lake

Unity Catalog

Azure Data Lake Storage Gen2

Azure Monitor

Azure AD

Key Vault

Azure DevOps

Job description

Position: Senior Azure Databricks Engineer

Working Location: Glendale AZ

Experience: 8 to10 years

Roles & Responsibilities

• Collaborate with Data Platforms and Integration team in architecting and implementing scalable enterprise data platform solutions utilizing Azure Databricks, Delta Lake, Unity Catalog, Azure Data Lake Storage Gen2, and related Azure services.
• Design and develop enterprise data engineering frameworks supporting batch, streaming, ELT/ETL, and real-time analytics workloads.
• Administer and maintain Azure Databricks workspaces including cluster configuration, job scheduling, workspace administration, Unity Catalog implementation, secret management, compute policies, cluster policies, and workspace governance.
• Define and implement enterprise security architecture including identity integration with Microsoft Entra ID (Azure AD), Role-Based Access Control (RBAC), Privileged Identity Management (PIM), data encryption, private networking, managed identities, Key Vault integration, and secure access patterns.
• Establish enterprise data governance standards utilizing Unity Catalog including data lineage, data quality, metadata management, catalog organization, access policies, and audit capabilities.
• Optimize Databricks workloads through performance tuning, Spark optimization, cluster sizing, workload isolation, caching strategies, partitioning, Delta optimization, Photon, and cost optimization best practices.
• Design enterprise monitoring, logging, operational dashboards, alerting, and platform health monitoring utilizing Azure Monitor, Log Analytics, Application Insights, and Databricks system tables.
• Develop reusable architecture patterns, reference implementations, development standards, naming conventions, coding standards, operational procedures, and platform documentation.
• Support migration of legacy enterprise data warehouses, ETL solutions, and reporting platforms to Azure Databricks and modern Lakehouse architecture.
• Provide technical support and troubleshooting for production incidents related to Azure Databricks, Spark workloads, data pipelines, and platform infrastructure.

Requirements

• At least 8-10 years of experience in designing and implementing enterprise data platforms;
• At least 5 years of hands-on experience hands-on Azure Databricks experience in enterprise environments;
• Experience with architecting modern Lakehouse solutions using Azure Databricks and Delta Lake;
• Experience in administering Azure Databricks workspaces, clusters, Unity Catalog, compute policies, job orchestration, and workspace security;
• Experience in developing data engineering solutions using Apache Spark (PySpark and/or Spark SQL);
• Experience in implementing enterprise data governance utilizing Unity Catalog, metadata management, data lineage, and fine-grained access controls;
• Experience designing and working with enterprise security architectures including RBAC, Managed Identities, Private Endpoints, Key Vault integration, encryption, and network security;
• Experience implementing enterprise data quality frameworks, monitoring, observability, and operational dashboards;
• Strong understanding of data modelling techniques including dimensional modelling, Data Vault, medallion architecture, and enterprise data warehouse concepts;
• Strong knowledge of Azure Data Lake Storage Gen2, Azure Key Vault, Azure Monitor, Microsoft Entra ID (Azure AD), and Azure networking;
• Strong SQL programming skills and proficiency in Python and PySpark development
• Strong experience implementing CI/CD pipelines utilizing Azure DevOps, automated testing, release management, and deployment automation