Principal Data Engineer — Databricks & Platform Lead

ATS Client; Cape Town; On-site; ZAR 900,000 - 1,500,000; Full time

Job summary

ATS Client in Cape Town seeks a Principal Data Engineer to lead production data pipelines on Databricks and elevate engineering standards. You will drive performance, cost optimisation, and design robust data models in collaboration with Finance and IT stakeholders.

You will mentor junior engineers, influence business stakeholders, and work with SAP S/4HANA data sets, Fivetran connectors and Power BI semantic modelling to deliver trusted data solutions.

Qualifications

• 3+ years hands-on Databricks experience in pipelines and optimisation.
• Familiar with Spark Declarative Pipelines (SDP) and Metric Views.
• Strong Python/SQL/Spark programming skills.
• Experience with data modelling, database design and query optimisation.
• Experience coaching junior developers.
• Ability to influence business stakeholders.
• Experience with SAP S/4HANA data sets, Fivetran (SaaS connectors/HVR) and Power BI modelling.

Responsibilities

• Designing and building robust, production‑grade data pipelines on Databricks, using SDP when appropriate.
• Optimising platform performance and cost by tuning compute, jobs, queries, storage and layout.
• Defining and maintaining consistent business metrics and a semantic layer for trusted reporting.
• Integrating data from core business systems (SAP S/4HANA) and other sources using Fivetran.
• Setting engineering standards across the team including CI/CD and data governance.
• Coaching and mentoring junior engineers and reviewing their work.
• Partnering with stakeholders to understand needs and deliver practical solutions.
• Engaging data owners to ensure data quality and fit for purpose.

Skills

Databricks

Python

SQL

Spark

coaching

Stakeholder management

SAP S/4HANA

Fivetran

Power BI semantic modelling

Tools

SDP

Metric Views

Job description

ATS Client in Cape Town seeks a Principal Data Engineer to lead production data pipelines on Databricks and elevate engineering standards. You will drive performance, cost optimisation, and design robust data models in collaboration with Finance and IT stakeholders.

You will mentor junior engineers, influence business stakeholders, and work with SAP S/4HANA data sets, Fivetran connectors and Power BI semantic modelling to deliver trusted data solutions.