Python Spark AI Engineer — Modern Data Platforms

WatersEdge Solutions; South Africa; Hybrid; ZAR 700,000 - 1,000,000; Full time

Benefits offered by this job

Home-office support

Wellness program

Share options

Learning opportunities

Competitive remuneration

Job summary

WatersEdge Solutions is seeking an experienced Python/Spark/AI Developer to modernise large-scale data platforms. You’ll migrate T-SQL workloads to Spark SQL and Delta Lake pipelines, build REST APIs, and contribute AI capabilities within a spec-driven, test-focused team.

The role emphasizes autonomous work within a hybrid/remote environment and a strong emphasis on type-safe development, testing, and documentation.

Qualifications

• 4+ years of professional Python development.
• 2+ years of production experience building Spark or distributed data pipelines.
• Strong Python 3.12 proficiency.
• Experience with type hints and static analysis (pyright, mypy).
• Experience with Pydantic models and data validation.

Responsibilities

• Build production-grade Spark and PySpark pipelines on Delta Lake.
• Migrate legacy T-SQL workloads into Spark SQL and PySpark pipelines.
• Develop and maintain REST APIs using FastAPI or equivalent.
• Write automated pytest suites including unit/integration tests.
• Follow ADRs, design docs, and CI/test gates.

Skills

Python development

Spark / PySpark

Delta Lake

FastAPI

pytest

Docker

Git & CI

T-SQL migration

Education

Bachelor’s degree

Tools

Pyright / mypy

Pydantic

uv / Poetry

Click

Spark

Delta Lake

Job description

WatersEdge Solutions is seeking an experienced Python/Spark/AI Developer to modernise large-scale data platforms. You’ll migrate T-SQL workloads to Spark SQL and Delta Lake pipelines, build REST APIs, and contribute AI capabilities within a spec-driven, test-focused team.

The role emphasizes autonomous work within a hybrid/remote environment and a strong emphasis on type-safe development, testing, and documentation.