Lakehouse Data Engineer - Azure Databricks & Medallion

PBT Group; Johannesburg; On-site; ZAR 900,000 - 1,500,000; Full time

Job summary

PBT Group is seeking an experienced Data Engineer to join a project focused on implementing a modern Lakehouse architecture using the Medallion framework. You will design and maintain data pipelines with Azure Databricks and ADF, enabling Bronze, Silver and Gold data layers for governed analytics and AI-ready data products.

You will work with a Data Scientist and other stakeholders to deliver reliable, scalable data products and business views that support Power BI reporting, data science use

Qualifications

• Hands-on experience with Azure Databricks and Lakehouse architectures.
• Experience implementing Medallion architecture (Bronze/Silver/Gold).
• Proficient in Python, PySpark and SQL for data engineering.
• Experience building data pipelines with Azure Data Factory.
• Knowledge of data modelling, governance and warehousing principles.
• Familiar with DevOps, Git versioning and CI/CD pipelines.

Responsibilities

• Design, develop and maintain scalable data pipelines using Azure Databricks and ADF.
• Ingest data from multiple sources and ensure reliable landing and processing.
• Support semantic modelling for Power BI reporting and AI use cases.
• Collaborate with Data Scientists and stakeholders to deliver governed data products.
• Deploy production data pipelines and maintain documentation.

Skills

Azure Databricks

Python

PySpark

SQL

Data Factory

Data modelling

Git

CI/CD

Apache Kafka

Apache Spark

Azure Functions

Power BI reporting

Tools

Databricks Unity Catalog

Azure Data Factory

Azure Functions

Apache Kafka

Apache Spark

Power BI

Job description

PBT Group is seeking an experienced Data Engineer to join a project focused on implementing a modern Lakehouse architecture using the Medallion framework. You will design and maintain data pipelines with Azure Databricks and ADF, enabling Bronze, Silver and Gold data layers for governed analytics and AI-ready data products.

You will work with a Data Scientist and other stakeholders to deliver reliable, scalable data products and business views that support Power BI reporting, data science use