Azure Databricks Lakehouse Admin

NYU Langone Hospitals; New York, Northern (NY, KY); Hybrid; USD 101,000 - 155,000; Full time

Job summary

NYU Langone Health is seeking an Azure Databricks Administrator to manage and secure the Databricks platform across workspaces, clusters, and jobs. You will oversee user access, runtime management, and policy enforcement while collaborating with data engineers, scientists, and analysts to optimize data workflows.

The role emphasizes cost control, data governance, and secure storage with Azure Key Vault and Data Factory, contributing to robust data solutions and insights within a leading academic

Qualifications

• Bachelor's degree in Computer science, information systems, or engineering.
• 7+ years of experience administering data platforms.
• 3+ years experience with Azure Databricks Lakehouse.
• Proficiency in Python/PySpark, PowerShell, or Bash.
• Experience automating administrative tasks and workflows.
• Knowledge of security best practices and compliance requirements.

Responsibilities

• Manage and maintain Azure Databricks Platform, workspaces, clusters, and jobs.
• Oversee user administration including access controls and permissions.
• Handle library installations, runtime management, and policy enforcement.
• Implement/analyze cost control measures and monitor platform usage and performance.
• Administer Unity Catalog for data governance and security.
• Collaborate with data engineers, data scientists, and analysts to optimize data workflows and pipelines.
• Manage Azure storage solutions, including Blob Storage and Data Lake Storage.
• Administer Azure Key Vault for secure storage of secrets and keys.
• Configure and manage Azure Data Factory for data integration and ETL processes.
• Implement and manage VNETs, firewalls, Azure policies and security best practices.
• Implement and manage Azure DevOps for CI/CD pipelines and version control.
• Implement best practices for automated deployment and continuous integration.
• Set up budgets and alerts to monitor Azure costs and spend.
• Configure alerts for proactive issue detection and resolution.
• Databricks Lakehouse Monitoring develop and execute ad hoc operational reports.

Skills

Python/PySpark

Automation

Communication skills

Security best practices

Education

Bachelor's degree in Computer Science / Information Systems / Engineering

Tools

Azure Databricks

Azure DevOps

Unity Catalog

Azure Data Factory

Blob Storage / Data Lake Storage

Job description

NYU Langone Health is seeking an Azure Databricks Administrator to manage and secure the Databricks platform across workspaces, clusters, and jobs. You will oversee user access, runtime management, and policy enforcement while collaborating with data engineers, scientists, and analysts to optimize data workflows.

The role emphasizes cost control, data governance, and secure storage with Azure Key Vault and Data Factory, contributing to robust data solutions and insights within a leading academic