Senior Data Engineer (Databricks & Cloud Data Platforms)

DVT; Cape Town; On-site; ZAR 900,000 - 1,500,000; Full time

Job summary

DVT, based in Capetown, is hiring a Senior Data Engineer with Databricks expertise to design and deliver modern, scalable data platforms for enterprise clients. You will work closely with architects, analysts, data scientists, and client stakeholders to create secure, high-performing analytics platforms across cloud environments.

You will lead design efforts for lakehouse architectures, implement governance, and mentor junior engineers while delivering end-to-end data solutions using PySpark,

Qualifications

• 8+ years in Data Engineering projects.
• 3+ years of hands-on Databricks delivery experience.
• Deep expertise in Python and PySpark.
• Strong understanding of Medallion and Lakehouse architectures.
• Experience building live Databricks dashboards using SQL.
• Proficient with Azure data services (Synapse, ADF, OneLake, Purview).
• Experience with AWS data platforms (Glue, S3, Redshift).
• Consulting or client-facing environment experience.
• Bachelor’s degree in computer science, engineering, information systems, or equivalent.

Responsibilities

• Design and implement scalable Databricks-based data platforms.
• Build ETL/ELT pipelines for batch and streaming workloads.
• Lead data migration to cloud-based lakehouse architectures.
• Develop data processing solutions using PySpark, Python, and SQL.
• Implement Delta Lake, Unity Catalog, and Purview governance.
• Design lakehouse and medallion architectures for enterprises.
• Optimize Spark workloads for performance, cost, and scalability.
• Create Databricks Workflows, Airflow, or similar orchestration.
• Collaborate with stakeholders to define requirements.
• Mentor junior engineers and contribute to best practices.
• Produce architecture artefacts and reusable assets.

Skills

Databricks

Cloud data platforms

Mentoring

Education

Bachelor's degree in CS/Engineering

Tools

Databricks

Azure

AWS

Delta Lake

Unity Catalog

Purview

Airflow

ADF

Job description

• Databricks Expertise (Essential)

DVT is one of Africa’s leading software development consultancies, partnering with top organisations across South Africa and globally to deliver innovative, enterprise-grade technology solutions.

At DVT, our engineers work on complex, high-impact projects using modern cloud and data technologies while collaborating with some of the strongest technical talent in the industry. We are passionate about continuous learning, technical excellence, and growing our people through mentorship, technical communities, sponsored events, and ongoing professional development.

We are looking for a Capetown based Senior Data Engineer with deep Databricks expertise to help design and deliver modern, scalable data platforms for enterprise clients.

Job Description: Senior Data Engineer (Databricks & Cloud Data Platforms)

Position Overview

As a Senior Data Engineer, you will play a critical consulting role in designing, building, and modernising enterprise data platforms, with a strong focus on Databricks lakehouse implementations across cloud environments.

You will work closely with solution architects, analysts, data scientists, and client stakeholders to deliver secure, scalable, and high-performing analytics platforms that drive real business impact. This is a hands-on technical role suited to someone passionate about solving complex data challenges and delivering best-in-class cloud-native solutions.

What You’ll Be Doing

• Design and implement scalable Databricks-based data platforms
• Build robust ETL / ELT pipelines for batch and streaming workloads
• Lead data migration and modernisation initiatives from legacy platforms to cloud-based lakehouse architectures
• Develop high-performance data processing solutions using PySpark, Python, and SQL
• Implement Delta Lake, Unity Catalog, and modern governance frameworks including Purview
• Design enterprise-grade lakehouse and medallion architectures
• Optimise Spark workloads for performance, scalability, and cost efficiency
• Build orchestration solutions using Databricks Workflows, ADF, Airflow, or similar tools in Azure/AWS
• Collaborate with technical and business stakeholders to define solution requirements
• Mentor junior engineers and contribute to technical best practices within the Data & AI practice
• Produce technical documentation, architecture artefacts, and reusable delivery assets

Technical Requirements

• Databricks Expertise (Essential)
• Cloud Data Platform Experience (Azure Fabric, AWS) (Essential)

You are:

• A strong technical leader who can own complex delivery outcomes
• Comfortable engaging directly with clients and stakeholders
• A proactive problem-solver with excellent analytical skills
• Able to clearly communicate technical concepts to both technical and non-technical audiences
• Passionate about mentoring and uplifting engineering teams
• Delivery-focused, quality-driven, and adaptable in consulting environments

Minimum Experience

• 8+ years in Data Engineering projects
• 3+ years of hands-on Databricks delivery experience
• Deep expertise in Python and PySpark
• Strong understanding of the Medallion Architecture and Lakehouse Architecture
• Hands-on experience building and maintaining live Databricks AI/BI Governance Dashboards using SQL
• Strong experience with Azure data services including Synapse, ADF, OneLake, Purview
• Practical exposure to AWS data platforms including Glue, S3, Redshift
• Proven track record delivering data migration and modernisation projects
• Experience working in consulting or client-facing environments
• Bachelor’s degree in computer science, Engineering, Information Systems, or equivalent practical experience

Why Join DVT?

At DVT, you’ll have the opportunity to:

• Work on enterprise-scale data transformation projects
• Collaborate with top-tier engineering talent
• Expand your expertise across modern cloud and data platforms
• Grow through continuous learning, mentorship, and technical leadership opportunities
• Be part of a people-first engineering culture focused on innovation and excellence