Cape Town, South Africa | Posted on 11/11/2026
DatabricksConsultant – Build and Optimise Production Lakehouse Pipelines
We’relooking for a hands-on Databricks Consultant who designs, builds and optimiseslakehouse data pipelines and solutions on the Databricks platform.
Thisis a role for someone who likes getting into the platform and making datapipelines reliable, fast and cost-efficient. You’ll work across ingestion,transformation and delivery of curated data products using PySpark, Spark SQLand Delta Lake, together with job orchestration, performance and cost tuning,data quality and platform governance.
What you’ll do:
- Design,build and maintain production data pipelines for ingestion, transformation anddelivery of curated data.
- Developand optimise PySpark and Spark SQL transformations, including incremental andbatch processing patterns.
- Designand operate Delta Lake tables, including table layout, optimisation andfile-size management.
- Contributeto platform performance and cost efficiency through job, compute and querytuning.
- Implementdata quality checks, validation, testing and clear technical documentation.
- Buildand maintain workflow orchestration, job scheduling, retries and alerting forproduction pipelines.
- ApplyUnity Catalog for data cataloguing, permissions and lineage where in use.
- Followversion control and CI/CD practices for notebooks, packages, jobs and pipelineconfiguration.
- TuneSpark jobs for scale and cost, including partitioning, shuffles, caching andcluster configuration.
- Ingestand transform data from enterprise and SaaS sources into well-structuredmedallion (bronze, silver, gold) or equivalent layers.
- Troubleshootfailed jobs, slow queries and data defects, and implement lasting technicalcorrections.
- Packagereusable pipeline components, parameters and configuration for consistentdelivery.
- Documentpipeline design, table contracts, refresh behaviour and operational steps.
Requirements
What we’re looking for:
- Minimumtwo years of hands‑on, day-to-day Databricks experience.
- Strongproficiency in Python, PySpark and Spark SQL.
- Practicalexperience building and operationalising data pipelines on Databricks.
- Experienceworking with Delta Lake, including table design and routine optimisation.
- Solidgrasp of data modelling and the ability to write efficient, well-structured SQLand Spark transformations.
- Experiencewith incremental processing, orchestration and production operationalisation ofpipelines.
- Experienceon at least one major cloud platform (Azure, AWS or GCP).
- Familiaritywith Databricks compute, jobs, workflows and SQL warehouses, includingperformance and cost awareness.
- Experiencewith Git-based version control and CI/CD for data workloads.
- Workingknowledge of data quality checks, testing, technical documentation and accesscontrol.