Python/Spark/AI Developer

Tumaini Consulting; Durban; On-site; ZAR 900,000 - 1,300,000; Full time

Job summary

Together Staffing group is seeking an experienced Python/Spark/AI Developer to replatform legacy T-SQL workloads onto Spark and Delta Lake, while building robust APIs and AI-enabled capabilities. The role suits an autonomous developer delivering production-quality code in a specification-driven environment.

The candidate will design and implement Spark pipelines, translate T-SQL logic to Spark SQL, and contribute to testing, documentation, and ADR-driven architecture.

Qualifications

• 4+ years of professional Python development.
• 2+ years of production Spark/PySpark experience.
• Strong Python 3.12 with type hints and Pydantic.
• Strong Spark/PySpark knowledge and Spark SQL expertise.
• Proficient in translating legacy T-SQL to Spark SQL and handling MERGE operations.
• Experience with Delta Lake or similar lakehouse tech (Iceberg/Hudi).
• Testing experience with pytest and CI/CD practices.
• REST API development experience, preferably with FastAPI.
• Experience with Docker-based development environments and Git workflows.
• Bachelor's degree in Computer Science, Engineering or equivalent.
• Ability to work independently from written specs and requirements.

Responsibilities

• Build production-grade Spark/PySpark pipelines with Delta Lake.
• Replatform legacy T-SQL logic into Spark SQL with parity.
• Develop robust, type-safe Python with strong testing.
• Create REST APIs using FastAPI or similar frameworks.
• Implement authentication, idempotency and reliable job-status semantics.
• Validate migrated pipelines against legacy systems with evidence of parity.
• Work from ADRs and design documents.
• Contribute tests and documentation alongside code changes.
• Engage in PR-driven development and CI processes.

Skills

Python

PySpark

SQL

Testing (pytest)

Type hints / Pydantic

CI/CD

Education

Bachelor's degree in CS/Engineering

Tools

Delta Lake

Spark

FastAPI

Docker

Git

Job description

Our client is seeking an experienced Python / Spark / AI Developer to help replatform legacy T-SQL workloads onto modern Spark and Delta Lake pipelines while developing robust APIs and AI-enabled capabilities. This intermediate-to-senior role is suited to a developer who is comfortable working autonomously in a specification-driven environment and delivering production-quality, well-tested code.

Key Responsibilities

• Build production-grade Spark/PySpark pipelines using Delta Lake.
• Replatform legacy T-SQL logic into Spark SQL while preserving functional parity.
• Develop modern, type-safe Python using strong testing and static-analysis practices.
• Build REST APIs using FastAPI or equivalent frameworks.
• Implement authentication, idempotency and reliable job-status semantics.
• Validate migrated pipelines against legacy systems and provide evidence of output parity.
• Work from design documents and Architecture Decision Records (ADRs).
• Contribute tests and documentation alongside code changes.
• Participate in Git-based, PR-driven development and CI processes.

Essential Requirements

• 4+ years' professional Python development experience.
• 2+ years' production experience building Spark/PySpark or comparable distributed data pipelines.
• Strong Python 3.12 experience, including type hints, Pydantic, modern packaging and static analysis.
• Strong Apache Spark/PySpark knowledge, including DataFrame API, Spark SQL, partitioning and performance tuning.
• Understanding of Spark driver/executor architecture and Spark Connect.
• Production experience with Delta Lake or equivalent lakehouse technologies such as Iceberg or Hudi.
• Strong SQL skills and the ability to translate legacy T-SQL logic into Spark SQL.
• Experience with MERGE operations, schema evolution, idempotent writes, hashing, surrogate keys and deduplication.
• Strong automated testing experience using pytest.
• REST API development experience, preferably with FastAPI.
• Experience with Docker-based development environments.
• Strong Git and CI/CD discipline, including linting, type-checking and automated testing.
• Bachelor's degree in Computer Science, Engineering or equivalent experience.
• Ability to work independently from written specifications and technical requirements.

Advantageous

• LLM integration and AI feature development.
• Experience with Ollama, vLLM, llama.cpp, LM Studio or hosted AI APIs.
• Data governance, PII protection and privacy engineering.
• Experience with Hive Metastore, Trino or Apache Ranger.
• Azure data technologies including ADLS Gen2, Synapse/ADF, Key Vault or Service Bus.
• Observability tools such as OpenTelemetry, OpenLineage or structured logging.
• Data migration and output-parity experience.
• Databricks Certified Developer for Apache Spark or equivalent certification.