Python/Spark or AI Developer

HRBaires; España; A distancia; EUR 72.000 - 107.000; Jornada completa

Descripción de la vacante

HRBaires in Europe (Remote) is seeking a mid-senior Data Engineer to migrate legacy SQL Server / Azure Synapse pipelines to Open-source Spark and Delta Lake. You will build PySpark pipelines and FastAPI services that run and report on pipeline jobs.

You will ensure parity with legacy outputs, add LLM features, and collaborate on design docs and tests, while aligning to South Africa hours (UTC+2) in a fully remote setup.

Formación

• Modern typed Python (3.10+) with type hints passing pyright/mypy and Pydantic models.
• 2+ years production Apache Spark / PySpark and Spark SQL.
• Based in Europe or South Africa, able to work South Africa hours (UTC+2).
• Experience with FastAPI, Docker, Delta Lake and cloud data workflows.

Responsabilidades

• Rewrite legacy T-SQL logic as typed PySpark pipelines.
• Validate parity between new PySpark pipelines and legacy outputs.
• Develop FastAPI services to run and report on pipeline jobs.
• Integrate provider‑neutral LLM features into data workflows and ensure tests/docs accompany changes.

Conocimientos

Python

PySpark

Spark SQL

Delta Lake

FastAPI

Docker

T-SQL

Pydantic

Type hints

Herramientas

Open-source Spark

Kubernetes

Azure Synapse

Git

Descripción del empleo

Location: Europe (Remote) — must work South Africa hours (UTC+2)
Employment: Full-time
Seniority: Mid-Senior Level
Salary: $80,000 – $120,000 (estimate; client hasn’t disclosed range)

About The Company

A fast-growing privacy-preserving data collaboration fintech (Series A, backed by a global credit bureau and a major retailer), with teams spanning South Africa, the UK, the US, the UAE and Brazil. Banks and lenders use the platform to work with retailer and telco data without raw personal data changing hands — mainly to help people with no credit history access credit.

The role

The Data Team Is Migrating Legacy SQL Server / Azure Synapse Stored‑procedure Pipelines Onto Open‑source Apache Spark And Delta Lake. You Will

• Rebuild T‑SQL logic as typed, tested PySpark pipelines
• Prove the new pipelines match legacy outputs
• Build FastAPI services that run and report on pipeline jobs
• Add provider‑neutral LLM features (local and hosted models) into data workflows

Spec-first codebase: design docs and ADRs, with tests and documentation shipped alongside every change.

Core stack

Python, PySpark, Spark SQL, Delta Lake, FastAPI, pytest, Docker, T‑SQL, LLM integration

Required

• Modern typed Python (3.10+) as primary production language, with type hints passing pyright/mypy and Pydantic models — writes maintainable, tested code, not only notebooks
• 2+ years production Apache Spark / PySpark — DataFrame API and Spark SQL, including partitioning and performance tuning
• Based in Europe or South Africa, able to work South Africa hours (UTC+2)

Nice to have

• SQL Server / Synapse migration to Spark with validated output parity
• Open‑source or self‑managed Spark (Kubernetes, EMR, Dataproc, Spark Connect)
• FastAPI, LLM integration, privacy/data governance (PII tokenisation, GDPR/POPIA), fintech/banking/credit background

Skills: apache spark,credit,fastapi,pipelines,apache,azure,fintech,data,sql,python,spark,english