Databricks Administrator

CloudIngest

Dallas (TX)

On-site

USD 120,000 - 170,000

Full time

2 days ago

Be an early applicant

A complete application in a minute

Tailored resume and cover letter, ready to send.

Get past ATS filters

Job summary

CloudIngest INC seeks an experienced Databricks Administrator to manage security, optimization, and support for enterprise Databricks environments. You will handle workspace administration, Unity Catalog governance, RBAC, SSO, secrets, and cloud integrations across Azure, AWS, or GCP.

You will collaborate with data engineers, scientists, and security teams to deliver a secure, scalable data analytics platform, deploying via Terraform and CI/CD pipelines.

Qualifications

Experience administering Databricks workspaces across Dev, QA, UAT, and Prod environments.
Strong governance management with Unity Catalog and RBAC.
Experience integrating Databricks with identity providers (Azure AD, AWS IAM, Okta).
Ability to tune performance, costs, and autoscaling of Spark clusters.

Responsibilities

Install, configure, and administer Databricks workspaces and clusters.
Manage multiple environments and enforce cluster policies.
Operate SQL Warehouses and Databricks Jobs; monitor health and performance.
Implement RBAC, SSO, secrets, and Unity Catalog governance.
Automate provisioning and backups via Terraform and CI/CD.
Collaborate with data engineers, scientists, security, and cloud admins.
Ensure security, compliance, and cost optimization across platforms.

Skills

Databricks

Unity Catalog

RBAC

Azure

AWS

GCP

Terraform

Python

PowerShell

Git

Jenkins

GitHub Actions

Databricks SQL

Delta Lake

Spark

MLflow

Delta Live Tables

Tools

Terraform

Azure DevOps

GitHub Actions

Jenkins

CloudFormation

Job description

Job Summary

We are seeking an experienced Databricks Administrator to manage, secure, optimize, and support enterprise Databricks environments. The ideal candidate will have expertise in Databricks workspace administration, Unity Catalog, cluster management, security, performance optimization, automation, and cloud platforms such as Azure, AWS, or GCP. The role involves working closely with data engineers, data scientists, architects, security teams, and cloud administrators to deliver a secure and scalable data analytics platform.

Key Responsibilities

Databricks Administration

Install, configure, and administer Databricks workspaces.
Manage multiple Databricks environments (Development, QA, UAT, Production).
Configure and maintain clusters and cluster policies.
Manage SQL Warehouses and Databricks Jobs.
Monitor workspace health and performance.
Upgrade and maintain Databricks Runtime versions.
Optimize cluster utilization and reduce infrastructure costs.

Security & Governance

Implement Unity Catalog governance.
Configure Role-Based Access Control (RBAC).
Manage users, groups, service principals, and permissions.
Integrate Databricks with Azure AD, AWS IAM, or Okta.
Configure Single Sign-On (SSO).
Manage secrets using Databricks Secret Scopes or cloud key management services.
Ensure compliance with organizational security policies.

Workspace Administration

Create and manage workspaces.
Configure notebooks and repositories.
Manage Git integration.
Administer workspace permissions.
Manage cluster access policies.
Configure job permissions.
Manage libraries and dependencies.

Infrastructure Management

Deploy Databricks using Terraform.
Automate infrastructure provisioning.
Configure networking: VPC/VNet Private Link Secure Cluster Connectivity NSG/Security Groups Firewalls Configure storage accounts (ADLS, S3).
Manage DBFS and external storage.

Performance Optimization

Tune Spark clusters.
Optimize autoscaling.
Configure instance pools.
Monitor cluster performance.
Troubleshoot slow jobs.
Optimize notebook execution.
Manage caching strategies.

Monitoring & Troubleshooting

Monitor Databricks clusters.
Configure alerts.
Review audit logs.
Troubleshoot: Cluster failures Job failures Authentication issues Permission issues Connectivity issues Library conflicts Performance bottlenecks.

Automation

Automate workspace deployment.
Automate cluster creation.
Automate user provisioning.
Automate permission management.
Automate backup processes.
Automate scheduled jobs using Terraform, Azure DevOps, GitHub Actions, or Jenkins.

CI/CD