Azure Databricks Lakehouse Architect

NYU Langone Health; New York (NY); On-site; USD 101,000 - 155,000; Full time

Job summary

NYU Langone Health is seeking an Azure Databricks Administrator to manage Databricks platforms across workspaces, clusters, and jobs. The role focuses on user administration, security, and cost control, with collaboration across engineers, scientists, and analysts to optimize data workflows.

The candidate will implement CI/CD pipelines using Azure DevOps, automate tasks, and ensure secure, scalable data operations in a complex academic medical center environment.

Qualifications

• Bachelor's degree in Computer Science, Information Systems, or Engineering.
• 7+ years of related professional experience administering data platforms.
• 3+ years experience with Azure Databricks Lakehouse Platform.
• Proficiency in scripting languages such as Python/PySpark, PowerShell, or Bash.
• Experience in automating administrative tasks and workflows.
• Knowledge of security best practices and compliance requirements.

Responsibilities

• Manage and maintain Azure Databricks Platform, workspaces, clusters, and jobs.
• Oversee user administration including access controls and permissions.
• Handle library installations, runtime management, and policy enforcement.
• Implement/analyze cost control measures and monitor platform usage and performance.
• Administer Unity Catalog for data governance and security.
• Collaborate with data engineers, data scientists, and analysts to optimize data workflows and pipelines on Databricks.

Skills

Python/PySpark

PowerShell

Bash

Azure Databricks

Education

Bachelor's degree in Computer Science/Information Systems/Engineering

Tools

Azure Databricks

Delta Lake

Databricks SQL

Azure Data Factory

Job description

NYU Langone Health is seeking an Azure Databricks Administrator to manage Databricks platforms across workspaces, clusters, and jobs. The role focuses on user administration, security, and cost control, with collaboration across engineers, scientists, and analysts to optimize data workflows.

The candidate will implement CI/CD pipelines using Azure DevOps, automate tasks, and ensure secure, scalable data operations in a complex academic medical center environment.