Senior Software Engineer (f/m/d) - Databricks Data Engineering & Analytics (part-/full-time)

European Commodity Clearing AG; Leipzig; Hybrid; EUR 90.000 - 120.000; Vollzeit

Benefits dieser Stelle

Childcare assistance

Meal allowance

Job ticket

Sports and leisure events

Zusammenfassung

European Commodity Clearing AG is seeking a Senior Software Engineer (f/m/d) in Data Development to own Databricks data engineering and analytics initiatives. You will shape standards, raise capabilities, and mentor peers while building modern, cloud-native data pipelines.

You will work with Databricks on Google Cloud, apply best practices in Delta Lake and Lakehouse design, and collaborate with infrastructure, platform engineering, and DevOps to accelerate business impact.

Qualifikationen

• Proven experience building data pipelines in distributed systems.
• Strong Python/Scala coding skills and SQL proficiency.
• Hands-on Databricks production experience (1+ years) and cloud familiarity.

Aufgaben

• Design and implement production-grade DataOps pipelines on Databricks.
• Lead best practices adoption across the team and mentor colleagues.
• Collaborate with analytics and business stakeholders to translate requirements.
• Review code, run pair-programming sessions, and contribute to data governance.

Kenntnisse

Databricks

Python

SQL

Spark

Scala

Data engineering

Cloud data platforms

Code reviews

Tools

Databricks Workflows

Delta Lake

Unity Catalog

Google Cloud Platform

Lakehouse architecture

Jobbeschreibung

We’re flexible! We’re happy to receive applications in English or German.

This role as "Senior Software Engineer (f/m/d) - Databricks Data Engineering & Analytics" in our team "Data Development" is an opportunity to do more than contribute your technical expertise. You will take ownership of data engineering and play a key role in shaping the culture, standards, and practices of a growing team. As a catalyst for change, you will not simply fill an existing gap. You will raise the capabilities of the entire team and organization, creating an impact that is multiplicative rather than additive. You will work with Databricks from your first day, building on a modern, cloud-native platform instead of maintaining legacy systems. With dedicated teams responsible for infrastructure, platform engineering, and DevOps, you can concentrate on what you do best: advancing data engineering and mentoring your colleagues. The purpose behind this work is equally significant. ECC’s data is mission-critical for European energy markets, meaning your contribution will help ensure that homes stay warm and that Europe can successfully move towards a green future.

Our tools:

• Primary tech stack: Google Cloud Platform (Databricks), Apache Spark, Python, SQL, Delta Lake
• Orchestration: Databricks Workflows
• Governance: Unity Catalog
• Your platform is managed centrally by Deutsche Börse – you focus on data engineering and eventually on empowering business users, not on infrastructure or DevOps.
• Design and build production-grade data pipelines on Databricks (Google Cloud) using Databricks Workflows, Spark SQL, and Python. Focus on creating solutions that are fast to implement, cheap to maintain, and highly reliable.
• Lead the adoption of Databricks best practices across the team: Delta Lake optimization, Photon query acceleration, cost governance, and performance tuning. Document and teach these practices so your teammates become independent.
• Architect scalable lakehouse solutions that support both analytics and operational workloads, applying modern design patterns (medallion architecture, data mesh principles).
• Mentor and upskill the team in Databricks: conduct code reviews, pair-programming sessions, and knowledge-sharing workshops. Your technical depth should elevate the entire group.
• After all, we value the exchange of ideas, being there for each other and encouraging each other to become better. We call it: the exchange mindset.
• Databricks mastery: You have hands‑on production experience with Databricks (1+ years), including:
• Building and optimizing ETL/ELT pipelines with Apache Spark (PySpark or Scala), Lakeflow Connect, Lakeflow Pipelines
• Delta Lake table design, optimization, and troubleshooting
• Databricks Workflows for orchestration
• Performance tuning
• Cost optimization and monitoring
• Backend data engineering: Several years of professional experience building data pipelines in distributed systems. Python and/or Scala proficiency is essential; SQL expertise is critical.
• Modern tooling: Familiarity with Git-based version control, CI/CD principles, and testing frameworks for data code.
• Cloud comfort: You have worked with Google Cloud Platform (or AWS/Azure) and understand cloud data warehouse/lakehouse architecture.
• Analytical thinking: You can break down complex data problems, propose multiple solutions, and evaluate trade-offs (speed vs. cost vs. reliability).
• Communication: You can explain technical concepts to non-technical stakeholders (business analysts, product owners) and translate their requirements into data engineering solutions. Fluent in Business English and German at an intermediate level minimum.
• Attractive salary package with many advantages such as childcare, meal allowance, job ticket, sports and leisure events
• Flexible hybrid work concept and flexible working hours
• Personal development through extensive training opportunities
• A place in a dynamic and international team within EEX Group and Deutsche Börse Group
• A long-term perspective in the constantly growing and evolving energy industry