Data Platform Engineer - - Azure & Databricks

Watershed Consulting; Johannesburg; Hybrid; ZAR 900,000 - 1,200,000; Full time

Job summary

Watershed Consulting in Sandton, Johannesburg, seeks a Data Platform Engineer to design, build, and operate the Data Platform, blending infrastructure and data engineering to keep Databricks and Azure self-service for platform users.

You will implement CI/CD pipelines, manage Terraform IaC for Azure services, and collaborate with data engineers, security, and IT teams to deliver secure, observable platforms.

Qualifications

• Solid hands-on experience with Microsoft Azure infrastructure components (Entra ID, Key Vault, networking, storage).
• Practical experience with Azure Databricks, including Unity Catalog and cluster/job management.
• Proven experience authoring Terraform in multi-environment setups for reproducible deployments.
• Proficiency in Python; working knowledge of SQL. .NET experience is a strong advantage.
• Experience building and maintaining CI/CD pipelines using GitHub Actions, Azure DevOps, or similar tools.
• Good understanding of data platform governance, cost optimization, security controls, and monitoring practices.
• Strong troubleshooting skills across networking, authentication, and pipeline integrations.
• Excellent collaboration and documentation skills for cross-functional platform delivery.

Responsibilities

• Design, implement and maintain CI/CD pipelines (GitHub Actions, Azure DevOps or similar) for infrastructure and application deployments.
• Write and maintain Terraform infrastructure-as-code for Azure services and Azure Databricks, including Unity Catalog, workspaces, networking, and compute policies.
• Develop and support platform tooling and automation using Python and/or .NET.
• Build and maintain shared libraries or modules consumed by data engineers and analysts across the platform.
• Design and implement management pipelines and processes for platform concerns such as cost tracking, governance, provisioning, and monitoring.
• Collaborate with data engineers, analysts, security, and infrastructure teams to translate requirements into platform capabilities and secure designs.
• Troubleshoot infrastructure, pipeline, and integration issues, including networking, authentication, and connectivity problems.
• Contribute to platform documentation, runbooks, and best-practice standards to enable self-service for platform users.

Skills

Azure

Entra ID

Azure Databricks

Unity Catalog

Terraform

Python

.NET

SQL

GitHub Actions

Azure DevOps

CI/CD

Monitoring

Cost Management

Governance

Networking troubleshooting

Authentication & Authorization

Automation

Tools

Terraform

GitHub Actions

Azure DevOps

Job description

Job Title

Data Platform Engineer — Sandton, Johannesburg (3 days per week onsite)

Duration: 12 months (renewal)

Description

Purpose: Support the design, build, and operation of the Data Platform, blending infrastructure engineering with data engineering practices to keep the Databricks and Azure environment reliable, automated, and self-service for platform users.

Responsibilities

• Design, implement and maintain CI/CD pipelines (GitHub Actions, Azure DevOps or similar) for infrastructure and application deployments.
• Write and maintain Terraform infrastructure-as-code for Azure services and Azure Databricks, including Unity Catalog, workspaces, networking, and compute policies.
• Develop and support platform tooling and automation using Python and/or .NET.
• Build and maintain shared libraries or modules consumed by data engineers and analysts across the platform.
• Design and implement management pipelines and processes for platform concerns such as cost tracking, governance, provisioning, and monitoring.
• Collaborate with data engineers, analysts, security, and infrastructure teams to translate requirements into platform capabilities and secure designs.
• Troubleshoot infrastructure, pipeline, and integration issues, including networking, authentication, and connectivity problems.
• Contribute to platform documentation, runbooks, and best-practice standards to enable self-service for platform users.

Qualifications

• Solid hands-on experience with Microsoft Azure infrastructure components (Entra ID, Key Vault, networking, storage).
• Practical experience with Azure Databricks, including Unity Catalog and cluster/job management.
• Proven experience authoring Terraform in multi-environment setups for reproducible deployments.
• Proficiency in Python; working knowledge of SQL. .NET experience is a strong advantage.
• Experience building and maintaining CI/CD pipelines using GitHub Actions, Azure DevOps, or similar tools.
• Good understanding of data platform governance, cost optimization, security controls, and monitoring practices.
• Strong troubleshooting skills across networking, authentication, and pipeline integrations.
• Excellent collaboration and documentation skills for cross-functional platform delivery.

Skills

• Microsoft Azure
• Entra ID
• Azure Key Vault
• Azure Networking
• Azure Storage
• Azure Databricks
• Unity Catalog
• Terraform
• Infrastructure as Code
• Python
• .NET
• SQL
• GitHub Actions
• Azure DevOps
• CI/CD
• DevOps
• Monitoring
• Cost Management
• Governance
• Networking troubleshooting
• Authentication and Authorization
• Automation

. Skillset Required: Microsoft Azure, Entra ID, Azure Key Vault, Azure Networking, Azure Storage, Azure Databricks, Unity Catalog, Terraform, Infrastructure as Code, Python, .NET, SQL, GitHub Actions, Azure DevOps, CI/CD, DevOps, Monitoring, Cost Management, Governance, Networking troubleshooting, Authentication and Authorization, Automation