Databricks Production Data Pipeline Architect

Sabenza IT & Recruitment

Cape Town

On-site

ZAR 600,000 - 900,000

Full time

7 days ago
Be an early applicant

Get a reply from this employer

Your resume and cover letter, tailored to exactly what they’re hiring for.

Get past ATS filters

Job summary

Sabenza IT & Recruitment in Cape Town is seeking a hands-on Databricks Consultant to design, build and optimise production lakehouse pipelines on the Databricks platform. You will work across ingestion, transformation and delivery of curated data, focusing on reliability, speed and cost efficiency.

Key tasks include developing PySpark and Spark SQL transformations, building Delta Lake tables, implementing data quality checks, and maintaining workflow orchestration and CI/CD practices.

Qualifications

  • Hands-on Databricks experience with production pipelines.
  • Strong Python, PySpark and Spark SQL skills.
  • Experience modelling data and writing efficient SQL/Spark transformations.
  • Familiarity with Delta Lake and table design.
  • Experience with pipeline orchestration and CI/CD practices.

Responsibilities

  • Design, build and maintain production data pipelines for ingestion, transformation and delivery of curated data.
  • Develop and optimise PySpark and Spark SQL transformations, including incremental processing patterns.
  • Design and operate Delta Lake tables, including layout and optimisation.
  • Improve platform performance and cost efficiency through tuning.
  • Implement data quality checks, testing and documentation.
  • Build and maintain workflow orchestration, job scheduling, retries and alerts.
  • Apply governance, cataloguing and access control as needed.
  • Follow version control and CI/CD for notebooks, jobs and pipelines.
  • Tune Spark jobs for scale and cost, including partitioning and caching.
  • Ingest and transform data from enterprise sources into medallion layers.

Skills

Python
PySpark
Spark SQL
Data modelling
SQL
Cloud platforms

Tools

Databricks
Delta Lake
Git
CI/CD

Job description

Sabenza IT & Recruitment in Cape Town is seeking a hands-on Databricks Consultant to design, build and optimise production lakehouse pipelines on the Databricks platform. You will work across ingestion, transformation and delivery of curated data, focusing on reliability, speed and cost efficiency.

Key tasks include developing PySpark and Spark SQL transformations, building Delta Lake tables, implementing data quality checks, and maintaining workflow orchestration and CI/CD practices.

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Databricks Lakehouse Engineer: Build & Tune Pipelines
Databricks Lakehouse Engineer: Build & Tune Pipelines

Sabenza IT & Recruitment • Eersterivier

On-site
ZAR 900,000 - 1,300,000
Databricks Consultant
Databricks Consultant

Sabenza IT & Recruitment • Cape Town

On-site
ZAR 600,000 - 900,000
Databricks Consultant
Databricks Consultant

Sabenza IT & Recruitment • Eersterivier

On-site
ZAR 900,000 - 1,300,000
Databricks & Cloud Data Platform Lead — Cape Town
Databricks & Cloud Data Platform Lead — Cape Town

DVT • Cape Town

On-site
ZAR 900,000 - 1,500,000
Principal Data Engineer (Databricks)
Principal Data Engineer (Databricks)

Sabenza IT & Recruitment • Cape Town

On-site
ZAR 1,200,000 - 1,700,000
Senior AWS Data Engineer — Databricks & Lakehouse Expert
Senior AWS Data Engineer — Databricks & Lakehouse Expert

PBT Group • Johannesburg

On-site
ZAR 900,000 - 1,300,000
Principal Data Engineer — Databricks & Platform Lead
Principal Data Engineer — Databricks & Platform Lead

Ats Client • Cape Town

On-site
ZAR 900,000 - 1,500,000
Senior Data Engineer - Databricks | Hybrid Parktown
Senior Data Engineer - Databricks | Hybrid Parktown

Sasso Consulting (Pty) Ltd • Johannesburg

Hybrid
ZAR 796,000 - 1,133,000
Senior AWS Data Engineer - Databricks & Lakehouse Lead
Senior AWS Data Engineer - Databricks & Lakehouse Lead

JVR Software • Johannesburg

On-site
ZAR 900,000 - 1,400,000
Senior AWS Data Engineer - Databricks & Lakehouse Architect
Senior AWS Data Engineer - Databricks & Lakehouse Architect

PBT Group • Johannesburg

On-site
ZAR 900,000 - 1,300,000