Azure Databricks Admin: Performance, Security & Automation

TechDigital Group; Cincinnati (OH); On-site; USD 110,000 - 160,000; Full time

Job summary

TechDigital Group is seeking an experienced Azure Databricks Administrator to deploy and manage scalable Databricks workspaces in a secure, compliant environment. You will optimize Spark workloads, manage Unity Catalog, and implement IaC-driven deployments across multiple environments.

The role requires strong Azure cloud skills, RBAC, and experience with CI/CD pipelines and monitoring tools to ensure cost efficiency and reliable analytics workloads.

Qualifications

• 5+ years of Azure Databricks administration and performance optimization.
• Expertise in Apache Spark, PySpark, SQL, and Scala.
• Hands-on experience with Databricks Unity Catalog, Delta Lake, and MLflow.
• Strong knowledge of Azure cloud services (Data Lake, Synapse, Key Vault).
• Experience in Infrastructure as Code (Terraform, Bicep, or ARM Templates).
• Proficiency in CI/CD pipeline automation using Azure DevOps, GitHub Actions, or Jenkins.
• Strong understanding of network security, identity management (AAD), and encryption best practices.

Responsibilities

• Deploy, configure, and manage Azure Databricks workspaces for scalable, cost-efficient operations.
• Administer clusters, jobs, notebooks, and workflows ensuring high availability and performance.
• Monitor and optimize compute resource utilization and autoscaling.
• Manage Databricks Runtime versions, libraries, and dependencies across environments.
• Implement Unity Catalog for fine-grained access control and data governance.
• Enforce RBAC and integrate with Azure Active Directory; ensure SOC 2, HIPAA, GDPR compliance.
• Tune Spark workloads and troubleshoot performance bottlenecks in ETL/ML pipelines.
• Automate workspace deployment and CI/CD pipelines; monitor with Azure Monitor, Prometheus, or Grafana.

Skills

Azure Databricks administration

Apache Spark / PySpark / SQL / Scala

Databricks Unity Catalog

Delta Lake

CI/CD automation

Infrastructure as Code (Terraform, Bce

RBAC / Azure AD / IAM

Azure cloud services

Troubleshooting / problem-solving

Documentation

Tools

Databricks REST API

Azure Monitor / Prometheus / Grafana

AKS / Kubernetes

GitHub Actions / Azure DevOps / Jenkins

Terraform / ARM Templates / Bicep

Delta Lake optimization

Job description

TechDigital Group is seeking an experienced Azure Databricks Administrator to deploy and manage scalable Databricks workspaces in a secure, compliant environment. You will optimize Spark workloads, manage Unity Catalog, and implement IaC-driven deployments across multiple environments.

The role requires strong Azure cloud skills, RBAC, and experience with CI/CD pipelines and monitoring tools to ensure cost efficiency and reliable analytics workloads.