Azure Databricks Platform Administrator

SilkRoad; New York (NY); On-site; USD 101,000 - 155,000; Full time

Benefits offered by this job

Comprehensive benefits

Wellness program

Job summary

NYU Langone Health in New York is seeking an experienced Azure Databricks Administrator to help maintain a secure, high-performing data platform across Databricks workspaces, clusters, and jobs.

You will automate tasks with Python/PySpark, implement CI/CD with Azure DevOps, manage Unity Catalog and storage, and collaborate with data engineers and scientists to deliver robust data solutions.

This role emphasizes strong communication, credentialing, and adherence to security best practices.

Qualifications

• Bachelor's degree in Computer science, Information systems, or Engineering.
• 7+ years of related professional experience administering data platforms.
• 3+ years experience with Azure Databricks Lakehouse Platform.
• Proficiency in Python/PySpark, PowerShell, or Bash.
• Experience in automating administrative tasks and workflows.
• Knowledge of security best practices and compliance requirements.
• Experience with ETL processes, data pipelines, and big data technologies.
• Experience with backup and restore procedures for Databricks and Azure services.
• Strong verbal and written communication skills.
• Ability to document processes, procedures, and configurations clearly.

Responsibilities

• Manage and maintain Azure Databricks Platform, workspaces, clusters, and jobs.
• Oversee user administration including access controls and permissions.
• Handle library installations, runtime management, and policy enforcement.
• Implement/analyze cost control measures and monitor platform usage and performance.
• Administer Unity Catalog for data governance and security.
• Collaborate with data engineers, data scientists, and analysts to optimize data workflows.
• Manage Azure storage solutions, including Blob Storage and Data Lake Storage.
• Administer Azure Key Vault for secure storage of secrets and keys.
• Configure and manage Azure Data Factory for data integration and ETL processes.
• Implement and manage VNETs, firewalls, Azure policies and security best practices.
• Implement and manage Azure DevOps for CI/CD pipelines and version control.
• Implement best practices for automated deployment and continuous integration.
• Set up budgets and alerts to monitor and control Azure costs and spend.
• Implement monitoring solutions for platform health and performance.
• Configure alerts for proactive issue detection and resolution.
• Databricks Lakehouse Monitoring develop and execute ad hoc operational reports.

Skills

Python/PySpark

PowerShell

Bash

Strong verbal and written skills

Documentation

Education

Bachelor's degree in Computer science, Information systems, or Engineering

Tools

Azure Databricks

Azure Data Factory

Azure Key Vault

Azure DevOps

Unity Catalog

Delta Lake

Job description

NYU Langone Health in New York is seeking an experienced Azure Databricks Administrator to help maintain a secure, high-performing data platform across Databricks workspaces, clusters, and jobs.

You will automate tasks with Python/PySpark, implement CI/CD with Azure DevOps, manage Unity Catalog and storage, and collaborate with data engineers and scientists to deliver robust data solutions.

This role emphasizes strong communication, credentialing, and adherence to security best practices.