Data Engineer - Databricks (f/m/d)

softwareXperts GmbH; Wien; Vor Ort; EUR 70.000 - 110.000; Vollzeit

Zusammenfassung

softwareXperts GmbH is seeking a senior data engineer to shape our data platform for an international insurance group. You will design, build and operate data pipelines on Databricks and contribute to lakehouse architectures.

You will collaborate with data scientists, analysts and the platform team, ensure data quality in regulated environments, and optimize Spark workloads for performance and cost while leveraging Azure cloud where applicable. English is the project language.

Qualifikationen

• At least 5 years of professional experience in data engineering.
• Solid hands-on experience with Databricks (Apache Spark, Delta Lake)
• Strong skills in Python/PySpark and SQL
• Experience building and operating data pipelines (batch and/or streaming)
• Experience with data modelling and lakehouse architectures is a plus
• Cloud experience (preferably Microsoft Azure) is a plus
• Excellent English skills (project language)

Aufgaben

• Design, build and operate data pipelines on Databricks.
• Data modelling and building lakehouse structures (Delta Lake).
• Connecting and integrating source systems.
• Ensuring data quality and governance in a regulated environment.
• Performance and cost optimisation of Spark workloads.
• Providing data for analytics and AI use cases.
• Collaboration with data scientists, analysts and the platform team.

Kenntnisse

Databricks

Python/PySpark

SQL

Tools

Delta Lake

Azure

Jobbeschreibung

In this project you will work in the data platform environment of an international insurance group

Must haves

• Databricks
• Python/PySpark
• SQL

Nice to have

Your Tasks

- Design, build and operate data pipelines on Databricks
- Data modelling and building lakehouse structures (Delta Lake)
- Connecting and integrating source systems
- Ensuring data quality and governance in a regulated environment
- Performance and cost optimisation of Spark workloads
- Providing data for analytics and AI use cases
- Collaboration with data scientists, analysts and the platform team

Requirements

- At least 5 years of professional experience in data engineering
- Solid hands-on experience with Databricks (Apache Spark, Delta Lake)
- Strong skills in Python/PySpark and SQL
- Experience building and operating data pipelines (batch and/or streaming)
- Experience with data modelling and lakehouse architectures is a plus
- Cloud experience (preferably Microsoft Azure) is a plus
- Excellent English skills (project language)

The content of this advertisement relates to an order from our customer company in the context of personnel placement. As a recruiter, we support you throughout the entire application process.