Databricks Consultant

Sabenza IT & Recruitment

Cape Town

On-site

ZAR 600,000 - 900,000

Full time

7 days ago
Be an early applicant

A complete application in a minute

Tailored resume and cover letter, ready to send.

Get past ATS filters

Job summary

Sabenza IT & Recruitment in Cape Town is seeking a hands-on Databricks Consultant to design, build and optimise production lakehouse pipelines on the Databricks platform. You will work across ingestion, transformation and delivery of curated data, focusing on reliability, speed and cost efficiency.

Key tasks include developing PySpark and Spark SQL transformations, building Delta Lake tables, implementing data quality checks, and maintaining workflow orchestration and CI/CD practices.

Qualifications

  • Hands-on Databricks experience with production pipelines.
  • Strong Python, PySpark and Spark SQL skills.
  • Experience modelling data and writing efficient SQL/Spark transformations.
  • Familiarity with Delta Lake and table design.
  • Experience with pipeline orchestration and CI/CD practices.

Responsibilities

  • Design, build and maintain production data pipelines for ingestion, transformation and delivery of curated data.
  • Develop and optimise PySpark and Spark SQL transformations, including incremental processing patterns.
  • Design and operate Delta Lake tables, including layout and optimisation.
  • Improve platform performance and cost efficiency through tuning.
  • Implement data quality checks, testing and documentation.
  • Build and maintain workflow orchestration, job scheduling, retries and alerts.
  • Apply governance, cataloguing and access control as needed.
  • Follow version control and CI/CD for notebooks, jobs and pipelines.
  • Tune Spark jobs for scale and cost, including partitioning and caching.
  • Ingest and transform data from enterprise sources into medallion layers.

Skills

Python
PySpark
Spark SQL
Data modelling
SQL
Cloud platforms

Tools

Databricks
Delta Lake
Git
CI/CD

Job description

Cape Town, South Africa | Posted on 11/11/2026

DatabricksConsultant – Build and Optimise Production Lakehouse Pipelines

We’relooking for a hands-on Databricks Consultant who designs, builds and optimiseslakehouse data pipelines and solutions on the Databricks platform.

Thisis a role for someone who likes getting into the platform and making datapipelines reliable, fast and cost-efficient. You’ll work across ingestion,transformation and delivery of curated data products using PySpark, Spark SQLand Delta Lake, together with job orchestration, performance and cost tuning,data quality and platform governance.

What you’ll do:

  • Design,build and maintain production data pipelines for ingestion, transformation anddelivery of curated data.
  • Developand optimise PySpark and Spark SQL transformations, including incremental andbatch processing patterns.
  • Designand operate Delta Lake tables, including table layout, optimisation andfile-size management.
  • Contributeto platform performance and cost efficiency through job, compute and querytuning.
  • Implementdata quality checks, validation, testing and clear technical documentation.
  • Buildand maintain workflow orchestration, job scheduling, retries and alerting forproduction pipelines.
  • ApplyUnity Catalog for data cataloguing, permissions and lineage where in use.
  • Followversion control and CI/CD practices for notebooks, packages, jobs and pipelineconfiguration.
  • TuneSpark jobs for scale and cost, including partitioning, shuffles, caching andcluster configuration.
  • Ingestand transform data from enterprise and SaaS sources into well-structuredmedallion (bronze, silver, gold) or equivalent layers.
  • Troubleshootfailed jobs, slow queries and data defects, and implement lasting technicalcorrections.
  • Packagereusable pipeline components, parameters and configuration for consistentdelivery.
  • Documentpipeline design, table contracts, refresh behaviour and operational steps.
Requirements

What we’re looking for:

  • Minimumtwo years of hands‑on, day-to-day Databricks experience.
  • Strongproficiency in Python, PySpark and Spark SQL.
  • Practicalexperience building and operationalising data pipelines on Databricks.
  • Experienceworking with Delta Lake, including table design and routine optimisation.
  • Solidgrasp of data modelling and the ability to write efficient, well-structured SQLand Spark transformations.
  • Experiencewith incremental processing, orchestration and production operationalisation ofpipelines.
  • Experienceon at least one major cloud platform (Azure, AWS or GCP).
  • Familiaritywith Databricks compute, jobs, workflows and SQL warehouses, includingperformance and cost awareness.
  • Experiencewith Git-based version control and CI/CD for data workloads.
  • Workingknowledge of data quality checks, testing, technical documentation and accesscontrol.
Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Databricks Consultant
Databricks Consultant

Sabenza IT & Recruitment • Eersterivier

On-site
ZAR 900,000 - 1,300,000
Databricks Production Data Pipeline Architect
Databricks Production Data Pipeline Architect

Sabenza IT & Recruitment • Cape Town

On-site
ZAR 600,000 - 900,000
Databricks Lakehouse Engineer: Build & Tune Pipelines
Databricks Lakehouse Engineer: Build & Tune Pipelines

Sabenza IT & Recruitment • Eersterivier

On-site
ZAR 900,000 - 1,300,000
Principal Data Engineer (Databricks)
Principal Data Engineer (Databricks)

Sabenza IT & Recruitment • Cape Town

On-site
ZAR 1,200,000 - 1,700,000
Data Engineer (MLOps / Analytics Focus)
Data Engineer (MLOps / Analytics Focus)

Blue Pearl PTY • Johannesburg

On-site
Confidential
Senior Data Engineer (Databricks & Cloud Data Platforms)
Senior Data Engineer (Databricks & Cloud Data Platforms)

DVT • Cape Town

On-site
ZAR 900,000 - 1,500,000
MLOps Data Engineer: Databricks Pipelines & Analytics
MLOps Data Engineer: Databricks Pipelines & Analytics

Blue Pearl PTY • Johannesburg

On-site
Confidential
Python / Spark / AI Developer
Python / Spark / AI Developer

Perssol • Cape Town

Hybrid
ZAR 700,000 - 1,200,000
Databricks & Cloud Data Platform Lead — Cape Town
Databricks & Cloud Data Platform Lead — Cape Town

DVT • Cape Town

On-site
ZAR 900,000 - 1,500,000
Data Engineer
Data Engineer

Parvana • Bellville

Hybrid
ZAR 700,000 - 1,100,000