Remote Python/Spark Data Engineer – AI & Lakehouse

WatersEdge Solutions; South Africa; Hybrid; ZAR 900,000 - 1,400,000; Full time

Benefits offered by this job

Flexible hybrid work

Remote working options

Wellness initiatives

Home-office support

Learning & development

Employee share ownership

Job summary

WatersEdge Solutions is seeking an experienced Python/Spark/AI Developer to join a technically driven team modernising large-scale data platforms in South Africa. You will replatform legacy T-SQL workloads into Spark and Delta Lake pipelines, and build the APIs and AI capabilities around them, primarily using Python with strong typing and tests.

You’ll work autonomously from written requirements and architecture records, contribute to REST APIs with FastAPI, and collaborate in a hybrid/remote

Qualifications

• Bachelor's degree in Computer Science, Engineering or a related discipline.
• Proven track record delivering production Python and Spark pipelines.
• Spark certification or equivalent credential is advantageous but not required.

Responsibilities

• Build production-grade Spark and PySpark pipelines on Delta Lake.
• Replatform legacy T-SQL workloads and stored-procedure-era logic into Spark SQL.
• Translate existing data logic accurately while maintaining expected business and data semantics.
• Write modern, type-hinted Python with strict static analysis.
• Develop and maintain automated pytest suites covering unit, integration and end-to-end testing.
• Validate migrated pipelines against legacy systems and provide clear parity and regression evidence.
• Build REST APIs using FastAPI or equivalent frameworks.
• Implement API request validation, authentication, idempotency and job-status semantics.
• Work within Docker-based development environments using Compose stacks.
• Follow GitHub flow and PR-driven development practices.
• Maintain linting, type-checking and automated test gates within CI pipelines.
• Work from design documents, technical specifications and ADRs.
• Document technical changes and decisions as part of the development process.
• Contribute to AI and LLM-enabled functionality around data workflows.
• Work independently within a hybrid or remote engineering environment.

Skills

Python development

Spark pipelines

PySpark

Type hints

pytest

REST APIs

Docker

CI/CD

Git

SQL to Spark

API design

Education

Bachelor's degree in CS/Engineering

Tools

Delta Lake

Spark SQL

Pyright

mypy

Pydantic

uvicorn

Poetry

Click

Docker Compose

Databricks Spark

Job description

WatersEdge Solutions is seeking an experienced Python/Spark/AI Developer to join a technically driven team modernising large-scale data platforms in South Africa. You will replatform legacy T-SQL workloads into Spark and Delta Lake pipelines, and build the APIs and AI capabilities around them, primarily using Python with strong typing and tests.

You’ll work autonomously from written requirements and architecture records, contribute to REST APIs with FastAPI, and collaborate in a hybrid/remote