Mid-Senior PySpark Data Engineer — AI & Privacy (Remote Europe/S Africa)

HRBaires

Portugal

Remote

EUR 72,000 - 107,000

Full time

11 days ago

Get a reply from this employer

Your resume and cover letter, tailored to exactly what they’re hiring for.

Get past ATS filters

Job summary

HRBaires is expanding a data engineering team and seeks a mid-senior Data Engineer to migrate legacy SQL Server/Azure Synapse pipelines to Apache Spark and Delta Lake. You will implement typed PySpark pipelines, validate outputs against legacy results, and deploy FastAPI services to monitor and report on jobs.

You will also integrate provider-neutral LLM features into data workflows, writing tests and docs alongside code changes. Remote work is available with alignment to SA hours.

Qualifications

  • Modern typed Python (3.10+) with type hints passing pyright/mypy and Pydantic models.
  • 2+ years production Apache Spark / PySpark with DataFrame API and Spark SQL.
  • Based in Europe or South Africa, able to work South Africa hours (UTC+2).

Responsibilities

  • Rebuild T‑SQL logic as typed, tested PySpark pipelines.
  • Prove the new pipelines match legacy outputs.
  • Build FastAPI services that run and report on pipeline jobs.
  • Add provider‑neutral LLM features into data workflows.

Skills

Python 3.10+
PySpark
Spark SQL
Delta Lake
FastAPI
pytest
Docker
T-SQL
LLM integration

Tools

Docker

Job description

HRBaires is expanding a data engineering team and seeks a mid-senior Data Engineer to migrate legacy SQL Server/Azure Synapse pipelines to Apache Spark and Delta Lake. You will implement typed PySpark pipelines, validate outputs against legacy results, and deploy FastAPI services to monitor and report on jobs.

You will also integrate provider-neutral LLM features into data workflows, writing tests and docs alongside code changes. Remote work is available with alignment to SA hours.

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Python/Spark or AI Developer
Python/Spark or AI Developer

HRBaires • Portugal

Remote
EUR 72,000 - 107,000
Senior Databricks Data Engineer - PySpark/SQL, Remote
Senior Databricks Data Engineer - PySpark/SQL, Remote

HumanIT Digital Consulting • Lisboa

Remote
EUR 26,000 - 34,000
15th month salary
Health insurance for family
Birthday off
+2
Remote Senior .NET Data Engineer – Spark, ETL, Data Lakes
Remote Senior .NET Data Engineer – Spark, ETL, Data Lakes

LUZA Group • Lisboa

Remote
EUR 40,000 - 65,000
Remote work whenever possible
Equipment provided
Benefits plan
Senior Data Engineer: Spark, PySpark & Azure
Senior Data Engineer: Spark, PySpark & Azure

Aubay • Portugal

Hybrid
EUR 40,000 - 60,000
Health Insurance
Feedback and Coffee system
Training Academy
+1
Senior Data Engineer — Spark, Delta Lake & Airflow (Remote)
Senior Data Engineer — Spark, Delta Lake & Airflow (Remote)

Optiwisers • Portugal

Hybrid
EUR 45,000 - 65,000
Competitive salary and benefits package
Career growth and continuous learning
Opportunity to work on complex data projects
+1
Senior Backend Engineer — Remote (Azure Synapse, PySpark)
Senior Backend Engineer — Remote (Azure Synapse, PySpark)

Match Profiler • Lisboa

Remote
EUR 50,000 - 80,000
Personalized support
Manager feedback
Team celebrations
+4
Senior Data Engineer (Azure & Databricks)
Senior Data Engineer (Azure & Databricks)

TP • Lisboa

On-site
EUR 70,000 - 90,000
Senior Data Engineer — AI‑Driven Pipelines (Remote)
Senior Data Engineer — AI‑Driven Pipelines (Remote)

Mindera • Portugal

Remote
EUR 60,000 - 90,000
Health insurance
Flexible working hours
Remote work in Portugal
+3
Remote Data Platform Engineer - Spark, SQL & Cloud
Remote Data Platform Engineer - Spark, SQL & Cloud

Bridge351 • Lisboa

Remote
EUR 65,000 - 90,000
Health and Life Insurance
100% Remote opportunities
Tech Visa Support
Databricks Data Engineer — Remote & AI-Driven Pipelines
Databricks Data Engineer — Remote & AI-Driven Pipelines

LUZA Group • Lisboa

Hybrid
EUR 42,000 - 68,000
Remote work wherever possible
Equipment delivery
Benefits plan
Top locations