Principal Data Engineer (Databricks)

Sabenza IT

Cape Town

Presencial

ZAR 1.100.000 - 1.700.000

Tempo integral

Há 11 dias

Receba uma resposta desta empresa empregadora

Seu currículo e carta de apresentação personalizados exatamente para o que estão contratando.

Passe pelos filtros ATS

Resumo da vaga

Sabenza IT in Cape Town, South Africa, is seeking a Principal Data Engineer to lead hands-on development of robust production data pipelines on Databricks and to raise engineering standards across the team. The role interfaces with business stakeholders, department heads, and data owners in Finance and IT to deliver trusted data products.

You will optimise the platform for performance and cost, define a semantic layer and metrics, and coach junior engineers while driving data quality and

Qualificações

  • Three+ years of hands-on Databricks experience.
  • Familiarity with Spark Declarative Pipelines (SDP) and Metric Views.
  • Strong programming skills in Python, SQL and Spark.
  • Experience in data modelling, database design and query optimisation.
  • Experience coaching junior engineers and influencing stakeholders.

Responsabilidades

  • Design and build production-grade data pipelines on Databricks.
  • Optimize platform for performance and cost, manage storage and layout.
  • Define and maintain consistent business metrics and a semantic layer.
  • Integrate data from SAP S/4HANA and other sources using Fivetran/HVR.
  • Set engineering standards across the team including CI/CD and data governance.
  • Coach and mentor junior and mid-level engineers; review work.
  • Engage with stakeholders to shape practical, high-value solutions.
  • Work with Finance and IT data owners to ensure data quality.

Conhecimentos

Databricks
Spark Declarative Pipelines (SDP)
Metric Views
Python
SQL
Spark
Data modelling
Database design
Query optimisation
Stakeholder management

Formação

Relevant tertiary qualification in Computer Science/Information Systems/Engineering or related field

Ferramentas

SAP S/4HANA
Fivetran
HVR
Power BI semantic modelling

Descrição da vaga

Reporting to the Data & Analytics Lead, the Principal Data Engineer is a hands‑on role, combining the building and optimization of production data pipelines with the technical leadership needed to raise engineering standards and develop the people around them. The role functions as a key interface between business stakeholders and department heads, the wider data and analytics team, and the owners of source systems and data in Finance and IT.

KEY RESPONSIBILITIES
  • Design and build robust, production‑grade data pipelines on Databricks, making appropriate use of current platform capabilities such as Spark Declarative Pipelines (SDP).
  • Work “under the hood” to optimise the platform for performance and cost tuning compute, jobs and queries, managing storage and table layout, and keeping platform spend under control.
  • Define and maintain consistent business metrics and a semantic layer (for example, using Metric Views) so that reporting is built on trusted, reusable definitions.
  • Integrate data from core business systems, including SAP S/4HANA, and other sources, using tools such as Fivetran (both SaaS connectors and HVR).
  • Set and uphold engineering standards across the team, including code quality, testing, documentation, CI/CD and data governance.
  • Coach and mentor junior and mid-level engineers, review their work and help them develop.
  • Partner with business stakeholders to understand their needs, shape practical solutions and ensure the platform delivers genuine value.
  • Engage data owners in Finance and IT to ensure data is well understood, valid and fit for purpose, and initiate data quality improvements where required.
REQUIRED SKILLS AND EXPERIENCE
  • At least three years of hands‑on, day‑to‑day Databricks experience, covering both pipeline development and under-the-hood performance and cost optimisation.
  • Up to date with recent platform developments, such as Spark Declarative Pipelines (SDP) and Metric Views.
  • Strong proficiency in programming languages commonly used in data engineering, such as Python, SQL and Spark.
  • Advanced experience with data manipulation, data modelling, database design and query optimisation.
  • Experience managing or coaching junior developers.
  • A track record of managing and influencing business stakeholders.
KEY COMPETENCIES
  • Combining deep, hands‑on engineering skill with sound judgement about cost, performance and long‑term maintainability.
  • Coaching, mentoring and raising the capability of less experienced engineers.
  • Collaborating, communicating confidently and influencing business stakeholders.
  • Breaking down complex technical concepts and explaining them simply to non‑technical audiences.
  • Staying current with a fast‑moving platform and bringing new capabilities into everyday practice.
  • Taking ownership and driving work independently, from concept through to production.
NICE TO HAVE
  • Experience working with SAP S/4HANA data sets.
  • Experience with Fivetran, including both SaaS connectors and HVR.
  • Power BI semantic modelling experience.
QUALIFICATIONS
  • A relevant tertiary qualification in Computer Science, Information Systems, Engineering or a related field, or equal?? practical experience
Obtenha sua revisão de currículo grátis e confidencial.

ou arraste e solte seu arquivo aqui.

Similar jobs

Vagas similares que valem a pena comparar

Principal Data Engineer (Databricks)
Principal Data Engineer (Databricks)

Sabenza IT & Recruitment • Cape Town

Presencial
ZAR 1.200.000 - 1.700.000
Principal Data Engineer (Databricks)
Principal Data Engineer (Databricks)

Ats Client • Cape Town

Presencial
ZAR 900.000 - 1.500.000
Principal Data Engineer — Databricks & Platform Lead
Principal Data Engineer — Databricks & Platform Lead

Ats Client • Cape Town

Presencial
ZAR 900.000 - 1.500.000
Senior Data Engineer: Spark, Databricks & Cloud Pipelines
Senior Data Engineer: Spark, Databricks & Cloud Pipelines

Boardroom Appointments • Sandton

Presencial
ZAR 900.000 - 1.500.000
Databricks Consultant
Databricks Consultant

Sabenza IT & Recruitment • Eersterivier

Presencial
ZAR 900.000 - 1.300.000
Intermediate Analytics Engineer
Intermediate Analytics Engineer

X, bigly labs • Johannesburg

Presencial
ZAR 600.000 - 900.000
Data Engineer
Data Engineer

Ats Client • Johannesburg

Presencial
ZAR 850.000 - 1.250.000
Senior Data Engineer
Senior Data Engineer

Sabenza IT • Gauteng

Presencial
ZAR 600.000 - 900.000
Senior Data Engineer
Senior Data Engineer

Future Fit • Johannesburg

Presencial
ZAR 900.000 - 1.300.000
Senior Analytics Engineer
Senior Analytics Engineer

X, bigly labs • Johannesburg

Presencial
ZAR 900.000 - 1.500.000