Senior Azure Databricks Engineer (ESS-424)

Environmental and Safety Solutions, Inc.

Glendale (AZ)

On-site

USD 140,000 - 180,000

Full time

6 hours ago
Be an early applicant

A complete application in a minute

Tailored resume and cover letter, ready to send.

Get past ATS filters

Job summary

Environmental and Safety Solutions, Inc. is seeking an experienced data platform engineer to build and manage enterprise data platforms on Azure Databricks in an on-site Glendale, AZ environment.

You will implement Delta Lake, Unity Catalog, and Gen2 storage, design scalable data pipelines, and enforce RBAC, PIM, and Key Vault security. You will drive data governance, performance tuning, and monitoring using Azure Monitor and Application Insights while collaborating with security and IT teams to

Qualifications

  • 8–10 years designing enterprise data platforms.
  • 5+ years hands-on Azure Databricks in large environments.
  • Experience with Lakehouse and Delta Lake.
  • Administration of Databricks workspaces, Unity Catalog, and security controls.
  • CI/CD pipelines using Azure DevOps and automated testing.

Responsibilities

  • Collaborate on scalable data platform solutions using Azure Databricks and Delta Lake.
  • Design data engineering frameworks for batch, streaming, ELT/ETL, and real-time analytics.
  • Administer Databricks workspaces, clusters, Unity Catalog, and secret management.
  • Define and implement enterprise security architecture (RBAC, PIM, Key Vault).
  • Establish data governance with Unity Catalog (data lineage, cataloging, access policies).
  • Tune Databricks performance, partitioning, caching, and cost optimization.
  • Monitor and log platform health with Azure Monitor, Log Analytics, and Application Insights.
  • Develop reusable architecture patterns, standards, and documentation.
  • Support migration of legacy warehouses to Lakehouse architecture and provide incident support.

Skills

Azure Databricks
Delta Lake
Unity Catalog
Azure Data Lake Gen2
PySpark
Spark SQL
SQL
Python
CI/CD with Azure DevOps
Cloud governance

Tools

Azure Monitor
Log Analytics
Application Insights
Key Vault
Azure Entra ID
Photon
Azure Databricks administration
Spark tuning
Delta Lake optimization

Job description

  • U.S. Citizenship required and meet security-clearance requirements.
  • Collaborate with Data Platforms and Integration team in architecting and implementing scalable enterprise data platform solutions utilizing Azure Databricks, Delta Lake, Unity Catalog, Azure Data Lake Storage Gen2, and related Azure services.
  • Design and develop enterprise data engineering frameworks supporting batch, streaming, ELT/ETL, and real-time analytics workloads.
  • Administer and maintain Azure Databricks workspaces including cluster configuration, job scheduling, workspace administration, Unity Catalog implementation, secret management, compute policies, cluster policies, and workspace governance.
  • Define and implement enterprise security architecture including identity integration with Microsoft Entra ID (Azure AD), Role-Based Access Control (RBAC), Privileged Identity Management (PIM), data encryption, private networking, managed identities, Key Vault integration, and secure access patterns.
  • Establish enterprise data governance standards utilizing Unity Catalog including data lineage, data quality, metadata management, catalog organization, access policies, and audit capabilities.
  • Optimize Databricks workloads through performance tuning, Spark optimization, cluster sizing, workload isolation, caching strategies, partitioning, Delta optimization, Photon, and cost optimization best practices.
  • Design enterprise monitoring, logging, operational dashboards, alerting, and platform health monitoring utilizing Azure Monitor, Log Analytics, Application Insights, and Databricks system tables.
  • Develop reusable architecture patterns, reference implementations, development standards, naming conventions, coding standards, operational procedures, and platform documentation.
  • Support migration of legacy enterprise data warehouses, ETL solutions, and reporting platforms to Azure Databricks and modern Lakehouse architecture.
  • Provide technical support and troubleshooting for production incidents related to Azure Databricks, Spark workloads, data pipelines, and platform infrastructure.
  • U.S. Citizenship required and meet security-clearance requirements.
  • U.S. Citizenship required and meet security-clearance requirements.
  • Collaborate with Data Platforms and Integration team in architecting and implementing scalable enterprise data platform solutions utilizing Azure Databricks, Delta Lake, Unity Catalog, Azure Data Lake Storage Gen2, and related Azure services.
  • Design and develop enterprise data engineering frameworks supporting batch, streaming, ELT/ETL, and real-time analytics workloads.
  • Administer and maintain Azure Databricks workspaces including cluster configuration, job scheduling, workspace administration, Unity Catalog implementation, secret management, compute policies, cluster policies, and workspace governance.
  • Define and implement enterprise security architecture including identity integration with Microsoft Entra ID (Azure AD), Role-Based Access Control (RBAC), Privileged Identity Management (PIM), data encryption, private networking, managed identities, Key Vault integration, and secure access patterns.
  • Establish enterprise data governance standards utilizing Unity Catalog including data lineage, data quality, metadata management, catalog organization, access policies, and audit capabilities.
  • Optimize Databricks workloads through performance tuning, Spark optimization, cluster sizing, workload isolation, caching strategies, partitioning, Delta optimization, Photon, and cost optimization best practices.
  • Design enterprise monitoring, logging, operational dashboards, alerting, and platform health monitoring utilizing Azure Monitor, Log Analytics, Application Insights, and Databricks system tables.
  • Develop reusable architecture patterns, reference implementations, development standards, naming conventions, coding standards, operational procedures, and platform documentation.
  • Support migration of legacy enterprise data warehouses, ETL solutions, and reporting platforms to Azure Databricks and modern Lakehouse architecture.
  • Provide technical support and troubleshooting for production incidents related to Azure Databricks, Spark workloads, data pipelines, and platform infrastructure.
Roles & Responsibilities
  • Collaborate with Data Platforms and Integration team in architecting and implementing scalable enterprise data platform solutions utilizing Azure Databricks, Delta Lake, Unity Catalog, Azure Data Lake Storage Gen2, and related Azure services.
  • Design and develop enterprise data engineering frameworks supporting batch, streaming, ELT/ETL, and real-time analytics workloads.
  • Administer and maintain Azure Databricks workspaces including cluster configuration, job scheduling, workspace administration, Unity Catalog implementation, secret management, compute policies, cluster policies, and workspace governance.
  • Define and implement enterprise security architecture including identity integration with Microsoft Entra ID (Azure AD), Role-Based Access Control (RBAC), Privileged Identity Management (PIM), data encryption, private networking, managed identities, Key Vault integration, and secure access patterns.
  • Establish enterprise data governance standards utilizing Unity Catalog including data lineage, data quality, metadata management, catalog organization, access policies, and audit capabilities.
  • Optimize Databricks workloads through performance tuning, Spark optimization, cluster sizing, workload isolation, caching strategies, partitioning, Delta optimization, Photon, and cost optimization best practices.
  • Design enterprise monitoring, logging, operational dashboards, alerting, and platform health monitoring utilizing Azure Monitor, Log Analytics, Application Insights, and Databricks system tables.
  • Develop reusable architecture patterns, reference implementations, development standards, naming conventions, coding standards, operational procedures, and platform documentation.
  • Support migration of legacy enterprise data warehouses, ETL solutions, and reporting platforms to Azure Databricks and modern Lakehouse architecture.
  • Provide technical support and troubleshooting for production incidents related to Azure Databricks, Spark workloads, data pipelines, and platform infrastructure.
Requirements
  • At least 8-10 years of experience in designing and implementing enterprise data platforms.
  • At least 5 years of hands-on experience hands-on Azure Databricks experience in enterprise environments.
  • Experience with architecting modern Lakehouse solutions using Azure Databricks and Delta Lake.
  • Experience in administering Azure Databricks workspaces, clusters, Unity Catalog, compute policies, job orchestration, and workspace security.
  • Experience in developing data engineering solutions using Apache Spark (PySpark and/or Spark SQL).
  • Experience in implementing enterprise data governance utilizing Unity Catalog, metadata management, data lineage, and fine-grained access controls.
  • Experience designing and working with enterprise security architectures including RBAC, Managed Identities, Private Endpoints, Key Vault integration, encryption, and network security.
  • Experience implementing enterprise data quality frameworks, monitoring, observability, and operational dashboards.
  • Strong understanding of data modelling techniques including dimensional modelling, Data Vault, medallion architecture, and enterprise data warehouse concepts.
  • Strong knowledge of Azure Data Lake Storage Gen2, Azure Key Vault, Azure Monitor, Microsoft Entra ID (Azure AD), and Azure networking.
  • Strong SQL programming skills and proficiency in Python and PySpark development.
  • Strong experience implementing CI/CD pipelines utilizing Azure DevOps, automated testing, release management, and deployment automation.
Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Senior Data Engineer
Senior Data Engineer

Confidentiel • Pittsburgh

Hybrid
USD 150,000 - 190,000
Cloud Platform Manager
Cloud Platform Manager

Confidentiel • Pittsburgh

Hybrid
USD 140,000 - 190,000
Azure Databricks Engineer (Dallas, TX)
Azure Databricks Engineer (Dallas, TX)

Cedent • Dallas (TX)

On-site
USD 120,000 - 150,000
Databricks Data Architect
Databricks Data Architect

NewVision • San Francisco (CA)

Hybrid
USD 140,000 - 190,000
Hybrid work model
Azure Databricks Admin
Azure Databricks Admin

Nityo Infotech • Connecticut

On-site
USD 120,000 - 150,000
Databricks Data Architect
Databricks Data Architect

NewVision Software • United States

Remote
USD 120,000 - 180,000
Databricks Data Engineer
Databricks Data Engineer

EXL Talent Acquisition Team • New York (NY)

On-site
USD 170,000 - 230,000
Cloud Engineer Azure Databricks
Cloud Engineer Azure Databricks

Turtle Trax S.A. • United States

Remote
USD 140,000 - 190,000
DataBricks Administrator
DataBricks Administrator

Mothership • Cincinnati (OH)

On-site
USD 120,000 - 160,000
Lead Databricks Data Engineer
Lead Databricks Data Engineer

EXL Talent Acquisition Team • New York (NY)

On-site
USD 180,000 - 240,000