Senior Data Engineer (Spark & Python Specialist)

WatersEdge Solutions; Cape Town; Remote; ZAR 1,000,000 - 1,400,000; Full time

Benefits offered by this job

Flexible working arrangements

Wellness support and home office reimbursement

Continuous learning opportunities

Competitive salary and recognition for performance

Supportive and inclusive team culture

Job summary

A leading data engineering company is seeking a Senior Data Engineer (Spark & Python Specialist) to work remotely in South Africa. The role involves building scalable data solutions and optimising Spark-based processing. Candidates should have 6+ years of experience in Spark/PySpark and strong skills in Python and T-SQL. Enjoy flexible working arrangements and a competitive salary with a supportive team culture focused on accountability and work-life balance.

Qualifications

• 6+ years of Spark / PySpark experience.
• Strong production-grade Python capability.
• Solid T-SQL skills with experience interpreting existing SQL logic.
• Experience with Azure Synapse Analytics, Data Factory.
• Delta Lake and Parquet expertise in high-volume environments.
• Docker experience and cloud-agnostic engineering standards.
• Proven mentoring and cross-team collaboration.
• Knowledge of security, compliance and data governance.

Responsibilities

• Optimise Spark-based processing through best practices.
• Build and maintain data pipelines using Python, PySpark, and Delta Lake.
• Refactor legacy SQL-based ETL logic into clean Python libraries.
• Develop medallion architecture (Bronze–Silver–Gold).
• Support code-first orchestration with Airflow, Dagster, or Python wrappers.
• Participate in code reviews and mentor junior engineers.
• Contribute to automated testing with Pytest.
• Collaborate with data scientists, analysts, and stakeholders.
• Lead initiatives to strengthen data engineering tooling and practices.
• Ensure security, compliance and governance across processes.

Skills

Spark

Python

PySpark

Delta Lake

T-SQL

Docker

Airflow

Education

Bachelor’s degree in Computer Science or related field

Tools

Azure Synapse Analytics

Data Factory

Data Factory

Delta Lake

Parquet

Docker

Airflow

Dagster

Job description

Location: Remote (South Africa)
Employment Type: Full-Time
Industry: Data Engineering | Cloud Platforms | Financial Services Technology

WatersEdge Solutions is partnering with a client to recruit a highly skilled Senior Data Engineer (Spark & Python Specialist). This is a strong opportunity for a technically advanced engineer who enjoys building scalable, high-performance data solutions in a modern cloud environment. The role is ideal for someone who thrives on optimisation, code-first engineering, and modernising legacy data logic into clean, portable, Python-centric solutions.

About the Role

As a Senior Data Engineer, you’ll serve as a key technical contributor within the engineering team, focused on building, maintaining, and optimising large‑scale data processing engines. You’ll work extensively with Spark, PySpark, Delta Lake, and cloud‑based lakehouse environments, helping shape a provider‑agnostic platform with strong engineering standards, portability, and performance at its core.

Key Responsibilities

• Optimise Spark‑based processing through best practices in memory management, shuffle tuning, and partitioning
• Build and maintain data pipelines using Python, PySpark, Delta Lake, and Parquet
• Refactor legacy SQL‑based ETL logic into modular, testable, maintainable Python libraries
• Build and optimise medallion architecture layers across Bronze, Silver, and Gold
• Support code‑first orchestration approaches using tools such as Airflow, Dagster, or Python‑based wrappers
• Participate in code reviews and mentor junior engineers in PySpark best practices
• Contribute to automated testing frameworks using Pytest
• Work closely with data scientists, analysts and business stakeholders to deliver fit‑for‑purpose solutions
• Lead initiatives that strengthen data engineering capability, tooling and technical best practice
• Ensure strong security, compliance and governance across engineering processes

What You’ll Bring

• Bachelor’s degree in Computer Science, Information Systems, Engineering, or a related field
• 6+ years of Spark / PySpark experience
• Strong production‑grade Python capability
• Solid T‑SQL skills with experience interpreting and migrating existing SQL logic
• Experience with Azure Synapse Analytics, Dedicated SQL Pools and Data Factory
• Hands‑on expertise with Delta Lake and Parquet in high‑volume environments
• Experience with Docker and open‑source, cloud‑agnostic engineering standards
• Strong collaboration skills and a proven track record modernising large‑scale data workloads
• Experience mentoring engineers and contributing to technical excellence across a team
• Strong understanding of security, compliance and data governance principles

Nice to Have

• Exposure to Microsoft Fabric
• Experience with code‑first orchestration tooling such as Airflow or Dagster
• Experience building reusable internal Python libraries and automated testing patterns
• Background in high‑scale cloud data platform modernisation

What’s On Offer

• Fully remote role based in South Africa
• Flexible working arrangements
• Wellness support and home office reimbursement
• Continuous learning opportunities
• Competitive salary, ESOP and recognition for performance
• Supportive and inclusive team culture focused on accountability and work‑life balance

Company Culture

This is a team that values transparency, accountability, inclusion and technical excellence. You’ll join an environment that supports strong engineering standards, continuous learning and collaborative problem‑solving while giving people the flexibility to do their best work in a sustainable way.