Senior Data Engineer (Spark & Python Specialist)

Hire Resolve

Johannesburg

Presencial

ZAR 279.000 - 558.000

Tempo integral

14 dias+

Uma candidatura completa em um minuto

Currículo e carta de apresentação personalizados, prontos para serem enviados.

Passe pelos filtros ATS

Benefícios oferecidos por esta vaga

Competitive salary based on experience

Resumo da vaga

A leading data collaboration platform in Johannesburg is seeking a Senior Cloud Data Engineer to lead the optimization of high-performance data processing engines. The ideal candidate will have strong expertise in Spark and Python, with a solid understanding of the Azure ecosystem, Delta Lake, and best practices in security and governance. This role offers competitive compensation based on experience. Interested candidates should apply by emailing their CV.

Qualificações

  • 6+ years of Spark/PySpark experience with optimization expertise.
  • Proficiency in building reusable Python libraries and automated testing.
  • Strong experience with Azure tools and services.

Responsabilidades

  • Act as the internal SME for Spark internals and manage performance.
  • Build cloud-agnostic pipelines using Python and Delta Lake.
  • Migrate complex SQL-based ETL into modular Python libraries.
  • Manage Medallion Architecture ensuring optimal storage performance.
  • Support transition to code-centric orchestration patterns.

Conhecimentos

Spark/PySpark experience
Production-grade Python
Azure Synapse
Delta Lake
Docker
T-SQL
Security & Governance

Formação

Bachelor’s degree in Computer Science, Information Systems, Engineering, or a related field

Descrição da vaga

An award-winning privacy-preserving data collaboration platform that enables companies to analyze and collaborate on consumer data to gain insights, build predictive models, and monetize data without sharing the raw data or compromising consumer privacy, is seeking a Senior Cloud Data Engineer who will be a lead technical contributor responsible for building and optimizing high-performance data processing engines.

Responsibilities
  • Spark Optimization: Act as the internal SME for Spark internals; manage memory, shuffle tuning, and partitioning for cost-effective performance.

  • Cloud-Agnostic Development: Build pipelines using Python and Delta Lake, decoupling code from specific cloud providers and reducing reliance on GUI tools (e.g., ADF).

  • Refactoring & Modernization: Migrate complex SQL-based ETL into modular, testable, and maintainable Python libraries.

  • Lakehouse Engineering: Manage Medallion Architecture (Bronze/Silver/Gold) using Delta Lake, ensuring storage performance via Z-Ordering and Vacuuming.

  • Code-First Orchestration: Support the transition to code-centric patterns (Airflow, Dagster) to prioritize portability.

  • Technical Excellence: Lead code reviews, mentor junior engineers, and implement automated testing frameworks (Pytest).

Minimum Requirements
  • Education: Bachelor’s degree in Computer Science, Information Systems, Engineering, or a related field.
  • Spark Mastery: 6+ years of Spark/PySpark experience; expert ability to diagnose bottlenecks via Spark UI and optimize complex DAGs.

  • Advanced Python: Proficiency in production-grade Python, including building reusable libraries and automated testing.

  • Azure Ecosystem: Strong experience with Azure Synapse, Dedicated SQL Pools, and Data Factory.

  • Modern Data Stack: Hands-on experience with Delta Lake, Parquet, and containerization (Docker).

  • Migration Skills: Solid T-SQL skills to interpret and migrate legacy logic into Python-centric environments.

  • Security & Governance: Proven ability to implement high levels of security and compliance across data processes.

Benefits
  • Competitive salary based on experience (salary can potentially be more based on experience/skills)

IFyou meet the above requirements and want to make a career-changing move, apply today by emailing your CV to itcareers@hireresolve.za.com

Obtenha sua revisão de currículo grátis e confidencial.

ou arraste e solte seu arquivo aqui.

Similar jobs

Vagas similares que valem a pena comparar

Senior Cloud Data Engineer - Remote
Senior Cloud Data Engineer - Remote

Hire Resolve • Cape Town

Remoto
ZAR 700.000 - 950.000
Senior Data Science Consultant
Senior Data Science Consultant

Hire Resolve • Johannesburg

Presencial
ZAR 1.200.000 - 1.800.000
Competitive salary based on experience
Senior Cloud Data Engineer — Spark & Python Lakehouse Lead
Senior Cloud Data Engineer — Spark & Python Lakehouse Lead

Hire Resolve • Johannesburg

Presencial
ZAR 279.000 - 558.000
Competitive salary based on experience
Senior Data Engineer: Aws Pipelines & Ai Governance
Senior Data Engineer: Aws Pipelines & Ai Governance

Jvr Software • África do Sul

Híbrido
ZAR 739.000 - 1.052.000
Hybrid work arrangement
Professional development opportunities
Health and retirement benefits
+1
Lead Data Engineer
Lead Data Engineer

Placements24 • Polokwane

Híbrido
ZAR 900.000 - 1.800.000
Medical insurance
Life insurance
Retirement fund
+3
Senior Data Engineer
Senior Data Engineer

Placements24 • Paarl

Híbrido
ZAR 900.000 - 1.200.000
Competitive salary and benefits
Health and wellness benefits
Retirement savings plan
+4
Senior Data Engineer
Senior Data Engineer

Placements24 • Vereeniging

Presencial
ZAR 700.000 - 1.100.000
Competitive salary package and bonus
Comprehensive health, dental, and退休/RE
Opportunities for professional develop
Data Engineering Lead
Data Engineering Lead

Blue Pearl PTY • Johannesburg

Presencial
ZAR 1.200.000 - 1.900.000
Senior Data Engineer - Johannesburg - up to R900k per annum
Senior Data Engineer - Johannesburg - up to R900k per annum

e-Merge IT Recruitment • Johannesburg

Híbrido
ZAR 810.000 - 990.000
Data Engineer Jhb - Northern Suburbs
Data Engineer Jhb - Northern Suburbs

Psybergate • Johannesburg

Presencial
ZAR 1.000.000 - 1.500.000