Senior Python & Spark AI Engineer - Delta Lake & APIs

Tumaini Consulting; Durban; Presencial; ZAR 900.000 - 1.300.000; Jornada completa

Descripción de la vacante

Together Staffing group is seeking an experienced Python/Spark/AI Developer to replatform legacy T-SQL workloads onto Spark and Delta Lake, while building robust APIs and AI-enabled capabilities. The role suits an autonomous developer delivering production-quality code in a specification-driven environment.

The candidate will design and implement Spark pipelines, translate T-SQL logic to Spark SQL, and contribute to testing, documentation, and ADR-driven architecture.

Formación

• 4+ years of professional Python development.
• 2+ years of production Spark/PySpark experience.
• Strong Python 3.12 with type hints and Pydantic.
• Strong Spark/PySpark knowledge and Spark SQL expertise.
• Proficient in translating legacy T-SQL to Spark SQL and handling MERGE operations.
• Experience with Delta Lake or similar lakehouse tech (Iceberg/Hudi).
• Testing experience with pytest and CI/CD practices.
• REST API development experience, preferably with FastAPI.
• Experience with Docker-based development environments and Git workflows.
• Bachelor's degree in Computer Science, Engineering or equivalent.
• Ability to work independently from written specs and requirements.

Responsabilidades

• Build production-grade Spark/PySpark pipelines with Delta Lake.
• Replatform legacy T-SQL logic into Spark SQL with parity.
• Develop robust, type-safe Python with strong testing.
• Create REST APIs using FastAPI or similar frameworks.
• Implement authentication, idempotency and reliable job-status semantics.
• Validate migrated pipelines against legacy systems with evidence of parity.
• Work from ADRs and design documents.
• Contribute tests and documentation alongside code changes.
• Engage in PR-driven development and CI processes.

Conocimientos

Python

PySpark

SQL

Testing (pytest)

Type hints / Pydantic

CI/CD

Educación

Bachelor's degree in CS/Engineering

Herramientas

Delta Lake

Spark

FastAPI

Docker

Git

Descripción del empleo

Together Staffing group is seeking an experienced Python/Spark/AI Developer to replatform legacy T-SQL workloads onto Spark and Delta Lake, while building robust APIs and AI-enabled capabilities. The role suits an autonomous developer delivering production-quality code in a specification-driven environment.

The candidate will design and implement Spark pipelines, translate T-SQL logic to Spark SQL, and contribute to testing, documentation, and ADR-driven architecture.