Remote Python & Spark AI Developer — EU/SA Hours

HRBaires; España; Remote; EUR 72,000 - 107,000; Full time

Job summary

HRBaires in Europe (Remote) is seeking a mid-senior Data Engineer to migrate legacy SQL Server / Azure Synapse pipelines to Open-source Spark and Delta Lake. You will build PySpark pipelines and FastAPI services that run and report on pipeline jobs.

You will ensure parity with legacy outputs, add LLM features, and collaborate on design docs and tests, while aligning to South Africa hours (UTC+2) in a fully remote setup.

Qualifications

• Modern typed Python (3.10+) with type hints passing pyright/mypy and Pydantic models.
• 2+ years production Apache Spark / PySpark and Spark SQL.
• Based in Europe or South Africa, able to work South Africa hours (UTC+2).
• Experience with FastAPI, Docker, Delta Lake and cloud data workflows.

Responsibilities

• Rewrite legacy T-SQL logic as typed PySpark pipelines.
• Validate parity between new PySpark pipelines and legacy outputs.
• Develop FastAPI services to run and report on pipeline jobs.
• Integrate provider‑neutral LLM features into data workflows and ensure tests/docs accompany changes.

Skills

Python

PySpark

Spark SQL

Delta Lake

FastAPI

Docker

T-SQL

Pydantic

Type hints

Tools

Open-source Spark

Kubernetes

Azure Synapse

Git

Job description

HRBaires in Europe (Remote) is seeking a mid-senior Data Engineer to migrate legacy SQL Server / Azure Synapse pipelines to Open-source Spark and Delta Lake. You will build PySpark pipelines and FastAPI services that run and report on pipeline jobs.

You will ensure parity with legacy outputs, add LLM features, and collaborate on design docs and tests, while aligning to South Africa hours (UTC+2) in a fully remote setup.