Senior Databricks Data Engineer: Spark, Delta Lake, Python

softwareXperts GmbH; Wien; Vor Ort; EUR 70.000 - 110.000; Vollzeit

Zusammenfassung

softwareXperts GmbH is seeking a senior data engineer to shape our data platform for an international insurance group. You will design, build and operate data pipelines on Databricks and contribute to lakehouse architectures.

You will collaborate with data scientists, analysts and the platform team, ensure data quality in regulated environments, and optimize Spark workloads for performance and cost while leveraging Azure cloud where applicable. English is the project language.

Qualifikationen

• At least 5 years of professional experience in data engineering.
• Solid hands-on experience with Databricks (Apache Spark, Delta Lake)
• Strong skills in Python/PySpark and SQL
• Experience building and operating data pipelines (batch and/or streaming)
• Experience with data modelling and lakehouse architectures is a plus
• Cloud experience (preferably Microsoft Azure) is a plus
• Excellent English skills (project language)

Aufgaben

• Design, build and operate data pipelines on Databricks.
• Data modelling and building lakehouse structures (Delta Lake).
• Connecting and integrating source systems.
• Ensuring data quality and governance in a regulated environment.
• Performance and cost optimisation of Spark workloads.
• Providing data for analytics and AI use cases.
• Collaboration with data scientists, analysts and the platform team.

Kenntnisse

Databricks

Python/PySpark

SQL

Tools

Delta Lake

Azure

Jobbeschreibung

softwareXperts GmbH is seeking a senior data engineer to shape our data platform for an international insurance group. You will design, build and operate data pipelines on Databricks and contribute to lakehouse architectures.

You will collaborate with data scientists, analysts and the platform team, ensure data quality in regulated environments, and optimize Spark workloads for performance and cost while leveraging Azure cloud where applicable. English is the project language.