Senior Cloud Data Engineer — Spark & Python Lakehouse Lead

Hire Resolve; Johannesburg; On-site; ZAR 279,000 - 558,000; Full time

Benefits offered by this job

Competitive salary based on experience

Job summary

A leading data collaboration platform in Johannesburg is seeking a Senior Cloud Data Engineer to lead the optimization of high-performance data processing engines. The ideal candidate will have strong expertise in Spark and Python, with a solid understanding of the Azure ecosystem, Delta Lake, and best practices in security and governance. This role offers competitive compensation based on experience. Interested candidates should apply by emailing their CV.

Qualifications

• 6+ years of Spark/PySpark experience with optimization expertise.
• Proficiency in building reusable Python libraries and automated testing.
• Strong experience with Azure tools and services.

Responsibilities

• Act as the internal SME for Spark internals and manage performance.
• Build cloud-agnostic pipelines using Python and Delta Lake.
• Migrate complex SQL-based ETL into modular Python libraries.
• Manage Medallion Architecture ensuring optimal storage performance.
• Support transition to code-centric orchestration patterns.

Skills

Spark/PySpark experience

Production-grade Python

Azure Synapse

Delta Lake

Docker

T-SQL

Security & Governance

Education

Bachelor’s degree in Computer Science, Information Systems, Engineering, or a related field

Job description

A leading data collaboration platform in Johannesburg is seeking a Senior Cloud Data Engineer to lead the optimization of high-performance data processing engines. The ideal candidate will have strong expertise in Spark and Python, with a solid understanding of the Azure ecosystem, Delta Lake, and best practices in security and governance. This role offers competitive compensation based on experience. Interested candidates should apply by emailing their CV.