Azure Databricks DevOps Engineer

Boardroom Appointments; Centurion; On-site; ZAR 600,000 - 900,000; Full time

Job summary

Boardroom Appointments is seeking an Azure Databricks DevOps Engineer to deploy and manage data and analytics platforms in Azure. You will automate infrastructure with Terraform, ARM, or Bicep, and build robust CI/CD pipelines using AzureDevOps, GitHub Actions, or Jenkins.

You will collaborate with Data Scientists, Engineers, and Analysts to optimize ETL processes, monitor environments with Azure Monitor, and ensure security and governance across cloud resources.

Qualifications

• Bachelor's or Master's degree in Computer Science, IT, or a related field.
• 5+ years in DevOps, Cloud Engineering, or Data Engineering roles.
• Strong hands-on experience with Azure Databricks, Azure Data Factory, and Azure Synapse Analytics.
• Proficiency with Terraform, ARM Templates, or Bicep for infrastructure automation.
• Experience with CI/CD tools such as Azure DevOps, GitHub Actions, or Jenkins.
• Knowledge of Python, Scala, or PowerShell for automation and scripting.

Responsibilities

• Databricks Management: Deploy, configure, and maintain Azure Databricks environments, ensuring scalability, security, and efficiency.
• Infrastructure Automation: Implement IaC using Terraform, ARM templates, or Bicep to automate cloud resource provisioning.
• CI/CD Pipelines: Develop and manage CI/CD pipelines using Azure DevOps, GitHub Actions, Jenkins to automate data pipeline deployments.
• Monitoring & Logging: Set up monitoring using Azure Monitor, Log Analytics, and Application Insights for proactive issue resolution.
• Security & Compliance: Implement RBAC, encryption, and network security policies in line with governance requirements.
• Data Integration & Optimization: Collaborate with Data Engineers to optimize ETL pipelines for performance and cost efficiency.

Skills

DevOps experience

Data Engineering

Agile/Scrum

Problem solving

Education

Bachelor's or Master's in CS/IT or related field

Tools

Azure Databricks

Azure Data Factory

Azure Synapse Analytics

Terraform

ARM Templates

Bicep

Azure DevOps

GitHub Actions

Jenkins

Python/Scala/PowerShell

Job description

About the job Azure Databricks DevOps Engineer

Key Responsibilities:

Databricks Management: Deploy, configure, and maintain Azure Databricks environments, ensuring scalability, security, and efficiency.

Infrastructure Automation: Implement and manage IaC using Terraform, ARM templates, or Bicep to automate cloud resource provisioning.

CI/CD Pipelines: Develop and manage CI/CD pipelines using tools like Azure DevOps, GitHub Actions, Jenkins to automate data pipeline deployments.

Monitoring & Logging: Set up and maintain monitoring solutions using Azure Monitor, Log Analytics, and Application Insights for proactive issue resolution.

Security & Compliance: Implement best practices for security, governance, and compliance, including RBAC, encryption, and network security policies.

Data Integration & Optimization: Work with Data Engineers to optimize ETL pipelines, ensuring high performance and cost efficiency.

Collaboration: Work closely with Data Scientists, Engineers, and Business Analysts to enable efficient data workflows.

Incident Management: Troubleshoot and resolve Databricks, Azure Kubernetes Services (AKS), and networking issues efficiently.

Documentation & Best Practices: Maintain technical documentation and establish best practices for Azure Databricks and DevOps processes.

Required Skills & Qualifications:

Bachelors or Masters degree in Computer Science, IT, or a related field.

5+ years of experience in DevOps, Cloud Engineering, or Data Engineering roles.

Strong hands-on experience with Azure Databricks, Azure Data Factory, and Azure Synapse Analytics.

Proficiency in Terraform, ARM Templates, or Bicep for infrastructure automation.

Experience with CI/CD tools such as Azure DevOps, GitHub Actions, or Jenkins.

Knowledge of Python, Scala, or PowerShell for automation and scripting.

Familiarity with containerization (Docker, Kubernetes, AKS).

Solid understanding of Azure networking, security, and identity management (IAM, RBAC, AAD).

Experience with monitoring and logging tools (Azure Monitor, Log Analytics, Prometheus, Grafana).

Strong problem-solving skills and ability to work in an Agile/Scrum environment.

Preferred Qualifications:

Azure Certifications (e.g., AZ-400, AZ-104, DP-203, DP-900).

Experience with Big Data processing frameworks (Apache Spark, Delta Lake).

Understanding of ML Ops and working with machine learning models in Databricks.

Knowledge of Event-driven architectures (Azure Event Hubs, Kafka, Service Bus).