Azure Databricks Admin

TechDigital Group; Cincinnati (OH); On-site; USD 110,000 - 160,000; Full time

Job summary

TechDigital Group is seeking an experienced Azure Databricks Administrator to deploy and manage scalable Databricks workspaces in a secure, compliant environment. You will optimize Spark workloads, manage Unity Catalog, and implement IaC-driven deployments across multiple environments.

The role requires strong Azure cloud skills, RBAC, and experience with CI/CD pipelines and monitoring tools to ensure cost efficiency and reliable analytics workloads.

Qualifications

• 5+ years of Azure Databricks administration and performance optimization.
• Expertise in Apache Spark, PySpark, SQL, and Scala.
• Hands-on experience with Databricks Unity Catalog, Delta Lake, and MLflow.
• Strong knowledge of Azure cloud services (Data Lake, Synapse, Key Vault).
• Experience in Infrastructure as Code (Terraform, Bicep, or ARM Templates).
• Proficiency in CI/CD pipeline automation using Azure DevOps, GitHub Actions, or Jenkins.
• Strong understanding of network security, identity management (AAD), and encryption best practices.

Responsibilities

• Deploy, configure, and manage Azure Databricks workspaces for scalable, cost-efficient operations.
• Administer clusters, jobs, notebooks, and workflows ensuring high availability and performance.
• Monitor and optimize compute resource utilization and autoscaling.
• Manage Databricks Runtime versions, libraries, and dependencies across environments.
• Implement Unity Catalog for fine-grained access control and data governance.
• Enforce RBAC and integrate with Azure Active Directory; ensure SOC 2, HIPAA, GDPR compliance.
• Tune Spark workloads and troubleshoot performance bottlenecks in ETL/ML pipelines.
• Automate workspace deployment and CI/CD pipelines; monitor with Azure Monitor, Prometheus, or Grafana.

Skills

Azure Databricks administration

Apache Spark / PySpark / SQL / Scala

Databricks Unity Catalog

Delta Lake

CI/CD automation

Infrastructure as Code (Terraform, Bce

RBAC / Azure AD / IAM

Azure cloud services

Troubleshooting / problem-solving

Documentation

Tools

Databricks REST API

Azure Monitor / Prometheus / Grafana

AKS / Kubernetes

GitHub Actions / Azure DevOps / Jenkins

Terraform / ARM Templates / Bicep

Delta Lake optimization

Job description

Azure Databricks Administration & Management:

• Deploy, configure, and manage Azure Databricks workspaces in a scalable, cost-efficient, and secure manner.
• Administer clusters, jobs, notebooks, and workflows, ensuring high availability and performance.
• Monitor and optimize compute resource utilization and autoscaling strategies to improve cost efficiency.
• Manage Databricks Runtime versions, libraries, and dependencies across environments.

Security & Compliance:

• Implement and manage Unity Catalog for fine-grained access control and data governance.
• Enforce Role-Based Access Control (RBAC) and integrate Databricks with Azure Active Directory (AAD).
• Ensure compliance with SOC 2, HIPAA, GDPR, and internal security standards.
• Set up audit logging, monitoring, and alerting for security and operational insights.

Performance Optimization & Troubleshooting:

• Tune Apache Spark workloads to improve query performance and resource efficiency.
• Analyze and troubleshoot performance bottlenecks in ETL and ML workloads.
• Optimize Delta Lake storage, caching, and indexing strategies for better query execution.

Automation & Infrastructure as Code (IaC):

• Automate Databricks workspace deployment using Terraform, ARM Templates, or Databricks REST API.
• Develop and maintain CI/CD pipelines for Databricks job deployment and configuration management.
• Implement monitoring solutions using Azure Monitor, Prometheus, or Grafana.

Collaboration & Integration:

• Work closely with data engineers, data scientists, and DevOps teams to support data pipelines and analytics workloads.
• Integrate Databricks with Azure Data Lake, Azure Synapse Analytics, and Snowflake.
• Provide technical guidance and best practices for efficient Spark job execution and cost optimization.

Required Skills & Experience:

• 5+ years of experience in Azure Databricks administration and performance optimization.
• Expertise in Apache Spark, PySpark, SQL, and Scala.
• Hands-on experience with Databricks Unity Catalog, Delta Lake, and MLflow.
• Strong knowledge of Azure cloud services (Azure Data Lake, Azure Synapse, Azure Key Vault, etc.).
• Experience in Infrastructure as Code (Terraform, Bicep, or ARM Templates).
• Proficiency in CI/CD pipeline automation using Azure DevOps, GitHub Actions, or Jenkins.
• Strong understanding of network security, identity management (AAD), and encryption best practices.
• Excellent problem-solving skills and ability to troubleshoot complex Databricks workloads.
• Strong communication and documentation skills.

Preferred Qualifications:

• Databricks Certified Associate or Professional certification.
• Experience with Azure Kubernetes Service (AKS) and serverless computing.
• Familiarity with Kafka, Apache Airflow, and event-driven architectures.
• Knowledge of Python, PowerShell, or Bash scripting for automation