Databricks Data Engineer

PBT Group; Cape Town; On-site; ZAR 500 000 - 700 000; Full time

Job summary

A leading company in Cape Town is seeking a Databricks Certified Data Engineer to enhance their data capabilities. The role involves designing and maintaining robust data pipelines, implementing workflows with Databricks and PySpark, and ensuring data quality. Candidates must have a strong background in the Azure Data ecosystem and relevant certifications. This is an excellent opportunity for those passionate about data engineering.

Qualifications

• 3+ years of experience in data engineering or related roles.
• Strong working knowledge of Databricks, PySpark, and Apache Spark.

Responsibilities

• Design, develop and maintain ETL/ELT pipelines using Azure Data Factory.
• Implement data processing workflows using Databricks and PySpark.
• Ensure high standards of data quality, reliability, and integrity.

Skills

Databricks

PySpark

SQL

Python

Data Engineering

Education

Databricks Data Engineer Associate or Professional Certification

Tools

Azure Data Factory

Terraform

Git

Job description

Databricks Data Engineer required in Cape Town.

We are looking for a Databricks Certified Data Engineer to join our Cape Town-based client on a resource augmentation contract.

The ideal candidate will have strong experience with the Azure Data ecosystem and advanced proficiency in Databricks and PySpark, with proven ability to build scalable, high-performance data pipelines and analytics solutions. Certification in Databricks is mandatory.

Duties and Responsibilities:

• Design, develop and maintain robust ETL/ELT pipelines using Azure Data Factory (ADF) and/or Fivetran
• Implement data processing workflows using Databricks, PySpark, and Delta Live Tables
• Build and optimise data transformation logic using Python and/or SQL
• Collaborate using Databricks notebooks and manage code versioning via Git
• Administer and monitor Azure-based environments, preferably with Terraform for infrastructure-as-code
• Apply data governance and lineage practices using Unity Catalog
• Work with structured and semi-structured data sourced from SAP systems
• Participate in data modelling, pipeline optimisation, and performance tuning
• Ensure high standards of data quality, reliability, and integrity

Requirements:

• Databricks Data Engineer Associate or Professional Certification (must be current/valid)
• 3+ years of experience in data engineering or related roles
• Strong working knowledge of Databricks, PySpark, and Apache Spark
• Proficiency in SQL and Python for data transformation and analysis
• Experience with Azure Data Factory and/or Fivetran
• Comfortable using Databricks Notebooks and Git for collaboration
• Experience with Terraform and Azure infrastructure administration
• Familiarity with Delta Lake, Delta Live Tables, and Unity Catalog
• Exposure to data ingestion or integration from SAP systems is advantageous