Python/Spark or AI Developer

HRBaires

Portugal

Teletrabalho

EUR 72 000 - 107 000

Tempo integral

Há 11 dias

Recebe uma resposta deste empregador

O teu currículo e carta de apresentação, adaptados exatamente ao que estão a contratar.

Ultrapassa os filtros ATS

Resumo da oferta

HRBaires is expanding a data engineering team and seeks a mid-senior Data Engineer to migrate legacy SQL Server/Azure Synapse pipelines to Apache Spark and Delta Lake. You will implement typed PySpark pipelines, validate outputs against legacy results, and deploy FastAPI services to monitor and report on jobs.

You will also integrate provider-neutral LLM features into data workflows, writing tests and docs alongside code changes. Remote work is available with alignment to SA hours.

Qualificações

  • Modern typed Python (3.10+) with type hints passing pyright/mypy and Pydantic models.
  • 2+ years production Apache Spark / PySpark with DataFrame API and Spark SQL.
  • Based in Europe or South Africa, able to work South Africa hours (UTC+2).

Responsabilidades

  • Rebuild T‑SQL logic as typed, tested PySpark pipelines.
  • Prove the new pipelines match legacy outputs.
  • Build FastAPI services that run and report on pipeline jobs.
  • Add provider‑neutral LLM features into data workflows.

Conhecimentos

Python 3.10+
PySpark
Spark SQL
Delta Lake
FastAPI
pytest
Docker
T-SQL
LLM integration

Ferramentas

Docker

Descrição da oferta de emprego

Location: Europe (Remote) — must work South Africa hours (UTC+2)
Employment: Full-time
Seniority: Mid-Senior Level
Salary: $80,000 – $120,000 (estimate; client hasn’t disclosed range)

About The Company

A fast-growing privacy-preserving data collaboration fintech (Series A, backed by a global credit bureau and a major retailer), with teams spanning South Africa, the UK, the US, the UAE and Brazil. Banks and lenders use the platform to work with retailer and telco data without raw personal data changing hands — mainly to help people with no credit history access credit.

The role

The Data Team Is Migrating Legacy SQL Server / Azure Synapse Stored‑procedure Pipelines Onto Open‑source Apache Spark And Delta Lake. You Will

  • Rebuild T‑SQL logic as typed, tested PySpark pipelines
  • Prove the new pipelines match legacy outputs
  • Build FastAPI services that run and report on pipeline jobs
  • Add provider‑neutral LLM features (local and hosted models) into data workflows

Spec-first codebase: design docs and ADRs, with tests and documentation shipped alongside every change.

Core stack

Python, PySpark, Spark SQL, Delta Lake, FastAPI, pytest, Docker, T‑SQL, LLM integration

Required
  • Modern typed Python (3.10+) as primary production language, with type hints passing pyright/mypy and Pydantic models — writes maintainable, tested code, not only notebooks
  • 2+ years production Apache Spark / PySpark — DataFrame API and Spark SQL, including partitioning and performance tuning
  • Based in Europe or South Africa, able to work South Africa hours (UTC+2)
Nice to have
  • SQL Server / Synapse migration to Spark with validated output parity
  • Open‑source or self‑managed Spark (Kubernetes, EMR, Dataproc, Spark Connect)
  • FastAPI, LLM integration, privacy/data governance (PII tokenisation, GDPR/POPIA), fintech/banking/credit background

Skills: apache spark,credit,fastapi,pipelines,apache,azure,fintech,data,sql,python,spark,english

Obtém a tua avaliação gratuita e confidencial do currículo.

ou arrasta e larga o ficheiro aqui.

Similar jobs

Ofertas semelhantes que vale a pena comparar

Mid-Senior PySpark Data Engineer — AI & Privacy (Remote Europe/S Africa)
Mid-Senior PySpark Data Engineer — AI & Privacy (Remote Europe/S Africa)

HRBaires • Portugal

Teletrabalho
EUR 72 000 - 107 000
Senior Data Engineer (Azure & Databricks)
Senior Data Engineer (Azure & Databricks)

TP • Lisboa

Presencial
EUR 70 000 - 90 000
Senior Data Engineer – Financial Services Data Lake (all genders) Porto, Lisbon
Senior Data Engineer – Financial Services Data Lake (all genders) Porto, Lisbon

Valantic • Porto

Híbrido
EUR 60 000 - 90 000
Data Engineer
Data Engineer

Intellias • Portugal

Presencial
EUR 70 000 - 100 000
Health Insurance
Annual Leave 22 days
Paid Sick Leave
+4
Remote Senior .NET Data Engineer – Spark, ETL, Data Lakes
Remote Senior .NET Data Engineer – Spark, ETL, Data Lakes

LUZA Group • Lisboa

Teletrabalho
EUR 40 000 - 65 000
Remote work whenever possible
Equipment provided
Benefits plan
Data Engineer (PySpark)
Data Engineer (PySpark)

Neotalent Conclusion • Lisboa

Presencial
EUR 35 000 - 65 000
Data Engineer
Data Engineer

Adecco Recruitment • Lisboa

Híbrido
EUR 60 000 - 100 000
Staff Data Engineer (Databricks specialist)
Staff Data Engineer (Databricks specialist)

Jobgether • Portugal

Teletrabalho
EUR 110 000 - 150 000
Fully remote work
International collaboration
Career growth
+1
Airline Ops Data Engineer - Hybrid & Ai Pipelines
Airline Ops Data Engineer - Hybrid & Ai Pipelines

Agile Engine • Viseu

Híbrido
EUR 45 000 - 70 000
Mentorship & Tech Talks
USD-based pay
Fortune 500 projects
+1
Senior Data Engineer (Remote From Portugal Or Selected Eu Countries)
Senior Data Engineer (Remote From Portugal Or Selected Eu Countries)

Fcamara • Portugal

Teletrabalho
EUR 60 000 - 90 000
Principais localizações