Databricks Lakehouse Engineer: Build & Tune Pipelines

Sabenza IT & Recruitment

Eersterivier

On-site

ZAR 900,000 - 1,300,000

Full time

7 days ago
Be an early applicant

A complete application in a minute

Tailored resume and cover letter, ready to send.

Get past ATS filters

Job summary

Sabenza IT & Recruitment is seeking a hands-on Databricks Consultant to design, build and optimise lakehouse data pipelines on the Databricks platform. You will work across ingestion, transformation and delivery of curated data products using PySpark, Spark SQL and Delta Lake, with emphasis on reliability, speed and cost efficiency.

You will design and operate pipelines, implement data quality checks, and optimise jobs for scale on cloud platforms.

Qualifications

  • Minimum two years of hands-on Databricks experience.
  • Strong proficiency in Python, PySpark and Spark SQL.
  • Practical experience building and operationalising data pipelines on Databricks.
  • Experience with Delta Lake, including table design and optimisation.
  • Familiarity with Git-based version control and CI/CD for data workloads.

Responsibilities

  • Design, build and maintain production data pipelines for ingestion, transformation and delivery of curated data.
  • Develop and optimise PySpark and Spark SQL transformations, including incremental and batch processing patterns.
  • Design and operate Delta Lake tables, including table layout, optimisation and file-size management.
  • Contribute to platform performance and cost efficiency through job, compute and query tuning.
  • Implement data quality checks, validation, testing and clear technical documentation.
  • Build and maintain workflow orchestration, job scheduling, retries and alerting for production pipelines.
  • Apply Unity Catalog for data cataloguing, permissions and lineage where in use.
  • Follow version control and CI/CD practices for notebooks, packages, jobs and pipeline configuration.
  • Tune Spark jobs for scale and cost, including partitioning, shuffles, caching and cluster configuration.
  • Ingest and transform data from enterprise and SaaS sources into medallion layers.
  • Troubleshoot failed jobs, slow queries and data defects, and implement lasting technical corrections.
  • Package reusable pipeline components, parameters and configuration for consistent delivery.
  • Document pipeline design, table contracts, refresh behaviour and operational steps.

Skills

Python
PySpark
Spark SQL
Delta Lake
Git
CI/CD
Data pipelines
Workflow orchestration

Job description

Sabenza IT & Recruitment is seeking a hands-on Databricks Consultant to design, build and optimise lakehouse data pipelines on the Databricks platform. You will work across ingestion, transformation and delivery of curated data products using PySpark, Spark SQL and Delta Lake, with emphasis on reliability, speed and cost efficiency.

You will design and operate pipelines, implement data quality checks, and optimise jobs for scale on cloud platforms.

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Databricks Production Data Pipeline Architect
Databricks Production Data Pipeline Architect

Sabenza IT & Recruitment • Cape Town

On-site
ZAR 600,000 - 900,000
Databricks Consultant
Databricks Consultant

Sabenza IT & Recruitment • Eersterivier

On-site
ZAR 900,000 - 1,300,000
Databricks Consultant
Databricks Consultant

Sabenza IT & Recruitment • Cape Town

On-site
ZAR 600,000 - 900,000
Senior AWS Data Engineer - Databricks & Lakehouse Architect
Senior AWS Data Engineer - Databricks & Lakehouse Architect

PBT Group • Johannesburg

On-site
ZAR 900,000 - 1,300,000
Senior AWS Data Engineer - Databricks & Lakehouse Lead
Senior AWS Data Engineer - Databricks & Lakehouse Lead

JVR Software • Johannesburg

On-site
ZAR 900,000 - 1,400,000
Senior AWS Data Engineer — Databricks & Lakehouse Expert
Senior AWS Data Engineer — Databricks & Lakehouse Expert

PBT Group • Johannesburg

On-site
ZAR 900,000 - 1,300,000
Lakehouse Data Engineer | Azure Databricks & AI-ready Data
Lakehouse Data Engineer | Azure Databricks & AI-ready Data

PBT Group • Johannesburg

On-site
ZAR 600,000 - 1,200,000
Senior Data Engineer: Spark, Databricks & Cloud Pipelines
Senior Data Engineer: Spark, Databricks & Cloud Pipelines

Boardroom Appointments • Sandton

On-site
ZAR 900,000 - 1,500,000
Databricks & Cloud Data Platform Lead — Cape Town
Databricks & Cloud Data Platform Lead — Cape Town

DVT • Cape Town

On-site
ZAR 900,000 - 1,500,000
Azure Data Engineer: Lakehouse & AI-Ready Data Pipelines
Azure Data Engineer: Lakehouse & AI-Ready Data Pipelines

PBT Group • Johannesburg

Hybrid
ZAR 700,000 - 950,000
Hybrid work arrangement
Johannesburg base