Senior Databricks Data Engineer | Hybrid (Cape Town)

Confluent; Wes-Kaap; Hybrid; ZAR 900,000 - 1,300,000; Full time

Benefits offered by this job

Office in Cape Town area

Medical aid

Retirement annuity

Bursaries

Insurance

Job summary

Confluent seeks a Senior Databricks Data Engineer to join our Hybrid team in Pinelands, Cape Town. You will design, build and evolve data pipelines across on-prem and cloud platforms, focusing on Azure Databricks and data security.

Work with senior engineers and data scientists on ML initiatives and data products. You will contribute to the architecture of the data platform, implement scalable Azure-based solutions, and collaborate across functions to meet data needs and ensure quality and

Qualifications

• Minimum 3 years experience as a data engineer, designing and implementing data pipelines.
• Practical experience in Azure services including Azure Data Factory and Databricks.
• Proven Python, PySpark and SQL/SparkSQL programming skills.
• 3+ years in IT roles focusing on data engineering, analytics or science.
• Knowledge of data storage solutions like SQL Server and Azure Data Lake Storage.
• Strong communication skills and team collaboration.

Responsibilities

• Contribute to the architecture and design of the Confluent Data Platform.
• Implement and support scalable data solutions on Azure, including Databricks.
• Develop and optimise production data pipelines for ingestion, processing and analytics.
• Ensure data security, quality, and integrity across pipelines.
• Collaborate with data product owners, analysts, data scientists and stakeholders.

Skills

3+ years data engineering

Python

PySpark

SQL

SparkSQL

Data engineering

Communication

Attention to detail

Problem solving

Data integration

Education

Bachelor's degree in Computer Science or related field

Azure Data Engineering / Databricks Certifications

Tools

Azure

Azure Data Factory

Azure Databricks

SQL Server

Azure Data Lake Storage

Job description

Confluent seeks a Senior Databricks Data Engineer to join our Hybrid team in Pinelands, Cape Town. You will design, build and evolve data pipelines across on-prem and cloud platforms, focusing on Azure Databricks and data security.

Work with senior engineers and data scientists on ML initiatives and data products. You will contribute to the architecture of the data platform, implement scalable Azure-based solutions, and collaborate across functions to meet data needs and ensure quality and