Senior Data Platform Engineer: Spark on Kubernetes

Ilias Solutions; Diegem; Hybrid; EUR 95,000 - 130,000; Full time

Job summary

ILIAS Solutions is seeking a senior backend engineer to own and operate a data runtime across multiple sites. You will manage Spark on Kubernetes, Delta Lake on MinIO, and PostgreSQL on Kubernetes, ensuring reliability even without direct access.

Responsibilities include schema design, data governance, and automation efforts across hybrid environments. You will develop backend services in Python and Java feeding dashboards and GenAI tooling, while leading migration strategies, write health

Qualifications

• 6+ years of backend engineering in production environments.
• 3+ years working with data-intensive systems.
• Proficiency in Python and SQL; Java working.
• Eligibility for national or NATO security clearance.

Responsibilities

• Own the full data runtime in production: deploy, configure, monitor and upgrade Spark on Kubernetes, Delta Lake on MinIO, and PostgreSQL on Kubernetes, including backup and restore and health checks.
• Build and operate Spark processing jobs end-to-end with Analytics Engineers, from schema design to diagnosis and Delta Lake hygiene.
• Design and steward PostgreSQL for analytical and serving workloads, including migrations and performance tuning.
• Deliver backend services in Python and Java that feed dashboards and GenAI assistant, with integration tests against real PostgreSQL and MinIO.
• Publish the full data stack as Helm charts and offline bundles, migrate from Docker Swarm to RKE2, and write clear runbooks for unreachable sites.
• Raise capability by mentoring engineers and setting standards for stateful service deployment and observability.

Skills

Spark on Kubernetes

PostgreSQL

Python

Java

Delta Lake

MinIO

Prometheus/JMX

Git

Tools

Kubernetes

Docker

pgBackRest

Barman

GitOps (Argo CD)

Prometheus

Longhorn

Job description

ILIAS Solutions is seeking a senior backend engineer to own and operate a data runtime across multiple sites. You will manage Spark on Kubernetes, Delta Lake on MinIO, and PostgreSQL on Kubernetes, ensuring reliability even without direct access.

Responsibilities include schema design, data governance, and automation efforts across hybrid environments. You will develop backend services in Python and Java feeding dashboards and GenAI tooling, while leading migration strategies, write health