Senior Databricks Data Engineer

Confluent; Wes-Kaap; Hybrid; ZAR 900,000 - 1,300,000; Full time

Benefits offered by this job

Office in Cape Town area

Medical aid

Retirement annuity

Bursaries

Insurance

Job summary

Confluent seeks a Senior Databricks Data Engineer to join our Hybrid team in Pinelands, Cape Town. You will design, build and evolve data pipelines across on-prem and cloud platforms, focusing on Azure Databricks and data security.

Work with senior engineers and data scientists on ML initiatives and data products. You will contribute to the architecture of the data platform, implement scalable Azure-based solutions, and collaborate across functions to meet data needs and ensure quality and

Qualifications

• Minimum 3 years experience as a data engineer, designing and implementing data pipelines.
• Practical experience in Azure services including Azure Data Factory and Databricks.
• Proven Python, PySpark and SQL/SparkSQL programming skills.
• 3+ years in IT roles focusing on data engineering, analytics or science.
• Knowledge of data storage solutions like SQL Server and Azure Data Lake Storage.
• Strong communication skills and team collaboration.

Responsibilities

• Contribute to the architecture and design of the Confluent Data Platform.
• Implement and support scalable data solutions on Azure, including Databricks.
• Develop and optimise production data pipelines for ingestion, processing and analytics.
• Ensure data security, quality, and integrity across pipelines.
• Collaborate with data product owners, analysts, data scientists and stakeholders.

Skills

3+ years data engineering

Python

PySpark

SQL

SparkSQL

Data engineering

Communication

Attention to detail

Problem solving

Data integration

Education

Bachelor's degree in Computer Science or related field

Azure Data Engineering / Databricks Certifications

Tools

Azure

Azure Data Factory

Azure Databricks

SQL Server

Azure Data Lake Storage

Job description

Position: Senior Databricks Data Engineer

(Hybrid/Pinelands, Cape Town)

Confluent Services, South Africa’s largest and multi-award winning debt management company, has a position available for aData Engineer.

This individual will join the Data Team and will be accountable for data engineering design, data ingestion and support across on-premise and cloud-based platforms.

What’s in it for you?

Being South Africa’s largest company of its kind, Confluent is continuously growing, with no signs of stopping. At Confluent, we build on current-generation platforms like Databricks and Microsoft Fabric, giving you hands‑on exposure to tools and patterns that are shaping the future of data engineering. This is not a legacy maintenance role, you’ll be involved in designing, building, and evolving data solutions from the ground up, with real room for creativity and engineering judgement.

You’ll work alongside a team of high caliber senior data engineers, where knowledge sharing is part of how we operate. Technical discussions are encouraged, opinions are respected, and decisions are driven by merit rather than hierarchy. Our team is actively involved in AI and machine learning initiatives, giving you exposure beyond traditional data pipelines.

We value clean architecture, thoughtful design, and creativity. You will have the opportunity to influence how things are built, contribute to architectural decisions, and help shape the evolution of our data platform.

Culturally, we are a diverse, respectful, and collaborative team. We invest in people, support continuous learning, and create an environment where engineers can grow both technically and professionally. Whether you want to deepen your engineering expertise or broaden your impact across the business, you’ll find room to do so here.

Beyond the work itself, Confluent offers a stable, growing environment where the data you build helps real South Africans improve their financial outcomes, giving purpose and impact to the systems you design.

Other key benefits

• We have a vibrant office in a lively part of the city, meaning you have easy access to stores and outlets if you need.
• We offer company benefits – medical aid, retirement annuity, Bursaries and Insurance
• We have a culture of growth, meaning you have access to move both upwards or across in the business if you want to.

Requirements

Candidates must be able to demonstrate:

• Minimum 3 years experience as a data engineer or in a similar role, designing and implementing data pipelines, data integration and ETL processes.
• Practical experience in Azure, Azure Data Factory and Azure Databricks is preferred.
• Proven experience inPython/PySparkandSQL/SparkSQLprogramming skills.
• At least 3 years of experience in Information Technology – specifically in technical data roles e.g. data engineering, data analytics or data science.
• Knowledge of data management and storage solutions, including SQL Server Database, Azure Data Lake Storage or similar.
• Having good communication skills and being able to operate in a team environment.
• Excellent attention to detail and problem-solving skills related to data, data structures and algorithms.

Key responsibilities will include:

• Contribute to the architecture and design of the Confluent Data Platform and ingestion patterns.
• Implement and support the scalable data solutions on the Microsoft Azure cloud platform, including Azure Databricks.
• Develop, maintain and optimise robust production data pipelines for data ingestion, processing, and transformation to support analytics, machine learning and other downstream data-driven services.
  • Ensure data security, quality and integrity across all of the pipelines and related data processes.
• Collaborate with the data product owner, analysts, data scientists, and business stakeholders to understand data requirements and implement effective data workflows.
• Assist in data initiatives and support technical teams including architects, analysts, software developers, and data scientists.

Qualifications:

• Bachelor's degree in Computer Science, Engineering, or a related field.
• Certifications in Azure, Microsoft Fabric Data Engineering, Azure Databricks Data Engineering or similar or equivalent practical experience.