Databricks Administrator

CloudIngest

Dallas (TX)

On-site

USD 110,000 - 170,000

Full time

3 hours ago

Be an early applicant

Application generator

Stand out for this role — generate a tailored resume and cover letter in about a minute.

Get past ATS filters

Job summary

CloudIngest is seeking an experienced Databricks Administrator to manage, secure, optimize, and support enterprise Databricks environments. You will administer workspaces, enforce governance with Unity Catalog, and manage clusters, jobs, and notebooks while integrating with cloud platforms like Azure, AWS, or GCP.

The role requires strong RBAC, security, and automation skills, with a focus on performance tuning, cost optimization, and scalable data analytics platform delivery.

Qualifications

Experience in managing Databricks workspaces and Unity Catalog.
Strong knowledge of RBAC, IAM, and security integration with cloud providers.
Proficient with IaC tooling (Terraform, ARM Templates, CloudFormation) and CI/CD.
Ability to optimize performance and control costs across multiple environments.

Responsibilities

Install, configure, and administer Databricks workspaces across DEV/QA/UAT/Prod.
Manage clusters, SQL Warehouses, jobs, and notebooks; monitor health.
Implement Unity Catalog governance and RBAC; integrate with IAM and SSO.
Automate provisioning with Terraform and CI/CD pipelines; ensure security and compliance.
Monitor performance, troubleshoot failures, and optimize costs; maintain backups.

Skills

Databricks Administration

Unity Catalog Governance

RBAC and IAM management

Cloud platforms (Azure, AWS, GCP)

Performance tuning and cost optim

Automation and IaC (Terraform, ARM, CF

CI/CD familiarity

Security and compliance

Monitoring and troubleshooting

Tools

Terraform

ARM Templates

CloudFormation

Azure DevOps

GitHub Actions

Jenkins

Git

Python

Bash

PowerShell

Okta

Azure AD

AWS IAM

Databricks API/CLI

DBFS

Job description

Job Summary

We are seeking an experienced Databricks Administrator to manage, secure, optimize, and support enterprise Databricks environments. The ideal candidate will have expertise in Databricks workspace administration, Unity Catalog, cluster management, security, performance optimization, automation, and cloud platforms such as Azure, AWS, or GCP. The role involves working closely with data engineers, data scientists, architects, security teams, and cloud administrators to deliver a secure and scalable data analytics platform.

Key Responsibilities

Databricks Administration

Install, configure, and administer Databricks workspaces.
Manage multiple Databricks environments (Development, QA, UAT, Production).
Configure and maintain clusters and cluster policies.
Manage SQL Warehouses and Databricks Jobs.
Monitor workspace health and performance.
Upgrade and maintain Databricks Runtime versions.
Optimize cluster utilization and reduce infrastructure costs.

Security & Governance

Implement Unity Catalog governance.
Configure Role-Based Access Control (RBAC).
Manage users, groups, service principals, and permissions.
Integrate Databricks with Azure AD, AWS IAM, or Okta.
Configure Single Sign-On (SSO).
Manage secrets using Databricks Secret Scopes or cloud key management services.
Ensure compliance with organizational security policies.

Workspace Administration

Create and manage workspaces.
Configure notebooks and repositories.
Manage Git integration.
Administer workspace permissions.
Manage cluster access policies.
Configure job permissions.
Manage libraries and dependencies.

Infrastructure Management

Deploy Databricks using Terraform.
Automate infrastructure provisioning.
Configure networking: VPC/VNet Private Link Secure Cluster Connectivity NSG/Security Groups Firewalls.
Configure storage accounts (ADLS, S3).
Manage DBFS and external storage.

Performance Optimization

Tune Spark clusters.
Optimize autoscaling.
Configure instance pools.
Monitor cluster performance.
Troubleshoot slow jobs.
Optimize notebook execution.
Manage caching strategies.

Monitoring & Troubleshooting

Monitor Databricks clusters.
Configure alerts.
Review audit logs.
Troubleshoot: Cluster failures Job failures Authentication issues Permission issues Connectivity issues Library conflicts Performance bottlenecks

Automation

Automate workspace deployment.
Automate cluster creation.
Automate user provisioning.
Automate permission management.
Automate backup processes.
Automate scheduled jobs using Terraform, Azure DevOps, GitHub Actions, or Jenkins.

CI/CD