Azure Databricks DevOps Engineer — Automation & Pipelines

Boardroom Appointments; Centurion; On-site; ZAR 600,000 - 900,000; Full time

Job summary

Boardroom Appointments is seeking an Azure Databricks DevOps Engineer to deploy and manage data and analytics platforms in Azure. You will automate infrastructure with Terraform, ARM, or Bicep, and build robust CI/CD pipelines using AzureDevOps, GitHub Actions, or Jenkins.

You will collaborate with Data Scientists, Engineers, and Analysts to optimize ETL processes, monitor environments with Azure Monitor, and ensure security and governance across cloud resources.

Qualifications

• Bachelor's or Master's degree in Computer Science, IT, or a related field.
• 5+ years in DevOps, Cloud Engineering, or Data Engineering roles.
• Strong hands-on experience with Azure Databricks, Azure Data Factory, and Azure Synapse Analytics.
• Proficiency with Terraform, ARM Templates, or Bicep for infrastructure automation.
• Experience with CI/CD tools such as Azure DevOps, GitHub Actions, or Jenkins.
• Knowledge of Python, Scala, or PowerShell for automation and scripting.

Responsibilities

• Databricks Management: Deploy, configure, and maintain Azure Databricks environments, ensuring scalability, security, and efficiency.
• Infrastructure Automation: Implement IaC using Terraform, ARM templates, or Bicep to automate cloud resource provisioning.
• CI/CD Pipelines: Develop and manage CI/CD pipelines using Azure DevOps, GitHub Actions, Jenkins to automate data pipeline deployments.
• Monitoring & Logging: Set up monitoring using Azure Monitor, Log Analytics, and Application Insights for proactive issue resolution.
• Security & Compliance: Implement RBAC, encryption, and network security policies in line with governance requirements.
• Data Integration & Optimization: Collaborate with Data Engineers to optimize ETL pipelines for performance and cost efficiency.

Skills

DevOps experience

Data Engineering

Agile/Scrum

Problem solving

Education

Bachelor's or Master's in CS/IT or related field

Tools

Azure Databricks

Azure Data Factory

Azure Synapse Analytics

Terraform

ARM Templates

Bicep

Azure DevOps

GitHub Actions

Jenkins

Python/Scala/PowerShell

Job description

Boardroom Appointments is seeking an Azure Databricks DevOps Engineer to deploy and manage data and analytics platforms in Azure. You will automate infrastructure with Terraform, ARM, or Bicep, and build robust CI/CD pipelines using AzureDevOps, GitHub Actions, or Jenkins.

You will collaborate with Data Scientists, Engineers, and Analysts to optimize ETL processes, monitor environments with Azure Monitor, and ensure security and governance across cloud resources.