Senior Azure Databricks Architect: Lakehouse & Security

Environmental and Safety Solutions, Inc.; Glendale (AZ); On-site; USD 120,000 - 180,000; Full time

Job summary

Environmental and Safety Solutions, Inc. is seeking an experienced Data Platform Engineer to architect and implement scalable enterprise data platforms on Azure Databricks.

Responsibilities include building batch, streaming, ELT/ETL, and real-time analytics workloads while administering Unity Catalog, security, and compute policies. You will design governance, optimize Spark workloads, and lead migration of legacy warehouses to a modern Lakehouse.

Qualifications

• 8–10 years of experience designing and implementing enterprise data platforms.
• 5+ years hands-on Azure Databricks experience in enterprise environments.
• Experience with architecting modern Lakehouse solutions using Azure Databricks and Delta Lake.
• Experience administering Azure Databricks workspaces, clusters, Unity Catalog, compute policies, job orchestration, and workspace security.
• Experience developing data engineering solutions using Apache Spark (PySpark and/or Spark SQL).
• Experience implementing Unity Catalog-based data governance with metadata, lineage, and access controls.
• Experience designing enterprise security architectures including RBAC, Private Endpoints, Key Vault integration, and network security.
• Experience implementing data quality frameworks, monitoring, and dashboards.
• Strong data modelling knowledge: dimensional, Data Vault, medallion architecture, and data warehouses.
• Strong knowledge of ADLS Gen2, Azure Key Vault, Azure Monitor, Entra ID, and networking.
• Strong SQL, Python, and PySpark development.
• Experience with CI/CD pipelines using Azure DevOps, testing, release management, deployment automation.

Responsibilities

• Collaborate with Data Platforms and Integration teams to architect scalable Azure-based data platforms.
• Design data engineering frameworks for batch, streaming, ELT/ETL, and real-time analytics workloads.
• Administer Databricks workspaces including clusters, Unity Catalog, and security configurations.
• Implement enterprise security using Azure AD, RBAC, PIM, encryption, and private networking.
• Establish Unity Catalog-based data governance with lineage, metadata management, and access policies.
• Optimize Databricks workloads with Spark tuning, partitioning, Delta optimization, Photon, and cost controls.
• Create monitoring and dashboards using Azure Monitor, Log Analytics, and Application Insights.
• Develop reusable architectures, coding standards, and platform documentation.
• Assist migration of legacy warehouses and ETL to Lakehouse on Databricks.
• Provide production support for Databricks, Spark workloads, data pipelines, and platform infra.

Skills

Azure Databricks

Delta Lake

Unity Catalog

Azure Data Lake Gen2

Spark (PySpark)

SQL

Python

Tools

Azure Databricks

Delta Lake

Unity Catalog

Azure Monitor

Key Vault

Azure DevOps

Job description

Environmental and Safety Solutions, Inc. is seeking an experienced Data Platform Engineer to architect and implement scalable enterprise data platforms on Azure Databricks.

Responsibilities include building batch, streaming, ELT/ETL, and real-time analytics workloads while administering Unity Catalog, security, and compute policies. You will design governance, optimize Spark workloads, and lead migration of legacy warehouses to a modern Lakehouse.