Cloud Data Platform Engineer – Azure & Databricks

Watershed Consulting; Johannesburg; Hybrid; ZAR 900,000 - 1,200,000; Full time

Job summary

Watershed Consulting in Sandton, Johannesburg, seeks a Data Platform Engineer to design, build, and operate the Data Platform, blending infrastructure and data engineering to keep Databricks and Azure self-service for platform users.

You will implement CI/CD pipelines, manage Terraform IaC for Azure services, and collaborate with data engineers, security, and IT teams to deliver secure, observable platforms.

Qualifications

• Solid hands-on experience with Microsoft Azure infrastructure components (Entra ID, Key Vault, networking, storage).
• Practical experience with Azure Databricks, including Unity Catalog and cluster/job management.
• Proven experience authoring Terraform in multi-environment setups for reproducible deployments.
• Proficiency in Python; working knowledge of SQL. .NET experience is a strong advantage.
• Experience building and maintaining CI/CD pipelines using GitHub Actions, Azure DevOps, or similar tools.
• Good understanding of data platform governance, cost optimization, security controls, and monitoring practices.
• Strong troubleshooting skills across networking, authentication, and pipeline integrations.
• Excellent collaboration and documentation skills for cross-functional platform delivery.

Responsibilities

• Design, implement and maintain CI/CD pipelines (GitHub Actions, Azure DevOps or similar) for infrastructure and application deployments.
• Write and maintain Terraform infrastructure-as-code for Azure services and Azure Databricks, including Unity Catalog, workspaces, networking, and compute policies.
• Develop and support platform tooling and automation using Python and/or .NET.
• Build and maintain shared libraries or modules consumed by data engineers and analysts across the platform.
• Design and implement management pipelines and processes for platform concerns such as cost tracking, governance, provisioning, and monitoring.
• Collaborate with data engineers, analysts, security, and infrastructure teams to translate requirements into platform capabilities and secure designs.
• Troubleshoot infrastructure, pipeline, and integration issues, including networking, authentication, and connectivity problems.
• Contribute to platform documentation, runbooks, and best-practice standards to enable self-service for platform users.

Skills

Azure

Entra ID

Azure Databricks

Unity Catalog

Terraform

Python

.NET

SQL

GitHub Actions

Azure DevOps

CI/CD

Monitoring

Cost Management

Governance

Networking troubleshooting

Authentication & Authorization

Automation

Tools

Terraform

GitHub Actions

Azure DevOps

Job description

Watershed Consulting in Sandton, Johannesburg, seeks a Data Platform Engineer to design, build, and operate the Data Platform, blending infrastructure and data engineering to keep Databricks and Azure self-service for platform users.

You will implement CI/CD pipelines, manage Terraform IaC for Azure services, and collaborate with data engineers, security, and IT teams to deliver secure, observable platforms.