Senior Data Engineer

Y-Axis; Berlin; Hybrid; EUR 90,000 - 150,000; Full time

Job summary

Y-Axis is seeking a senior data engineer to design and maintain robust data pipelines in a Databricks-centric environment. You will work with Delta Lake, Unity Catalog, and Lakeflow Declarative Pipelines to enable scalable data lakehouse architectures.

You will optimize PySpark workloads, write complex SQL, and monitor performance using Spark UI/ Photon, while ensuring data quality and governance with best practices and CI/CD on cloud platforms.

Qualifications

• 6+ years of data engineering experience.
• Last 3+ years using Databricks in production.
• Experience with Lakehouse architectures (Delta Lake, Unity Catalog).
• Strong Python, PySpark and SQL; proficient in Spark UI/Photon for performance diagnostics.

Skills

Data engineering

Databricks

Python

PySpark

SQL

Spark UI

Photon

Kafka Structured Streaming

CDC patterns

Data quality

Tools

Delta Lake

Unity Catalog

Lakeflow Declarative Pipelines

Databricks Asset Bundles

Terraform

Azure DevOps

GitHub Actions

Job description

• Requirements: 6+ years of data engineering experience, with at least the last three years on Databricks in production.
• Deep hands-on experience with medallion / Lakehouse architecture, Delta Lake, Unity Catalog, Lakeflow Declarative Pipelines and Databricks Asset Bundles.
• Strong python, PySpark and SQL, comfortable diagnosing performance issues on large workloads through Spark UI and Photon.
• Real production experience with streaming ingestion (Kafka Structured Streaming, Auto Loader) and CDC patterns.
• Experience implementing data quality at scale (Great Expectations or equivalent), plus lineage, cataloguing and access control.
• CI/CD on data platforms through Azure DevOps or GitHub Actions, with infrastructure as code (Terraform or equivalent).
• Fluent working English.