Senior Data Engineer – Lakehouse & Real-Time Pipelines

Huxley; Amsterdam; On-site; EUR 90,000 - 135,000; Full time

Benefits offered by this job

null

Job summary

Huxley is seeking a Senior Data Engineer to join a data-mesh driven team building a foundational customer data product. You will design and operate batch, streaming and near-real-time pipelines consolidating sales, payments, orders and gift-card data.

The role requires hands-on Databricks, Spark, Delta Lake and Unity Catalog expertise, strong Python/SQL skills and experience with governance, data-modeling and secure data delivery to analysts and business teams.

Qualifications

• Senior-level experience designing, building and operating production data platforms and data products.
• Hands-on Databricks experience including Spark, Delta Lake, Unity Catalog and Structured Streaming.
• Strong Python and advanced SQL skills.
• Experience with distributed data processing and scalable pipelines (batch and streaming).
• Data-modelling, data-quality validation, monitoring and alerting.
• Knowledge of event-driven architecture, schema evolution and error-handling patterns.
• Experience with Git, CI/CD and modern software engineering practices.
• Knowledge of data security, privacy, metadata and data lineage.

Responsibilities

• Design, build, test and operate batch, streaming and near-real-time data pipelines.
• Ingest and harmonise data from multiple source systems (sales, payments, orders, gift-card).
• Develop transformations and production data models using SQL, Python, BigQuery and dbt.
• Apply Databricks and Spark expertise to scalable processing and governance decisions.
• Implement data-quality validation, monitoring and automated checks.
• Mentor colleagues and collaborate with analysts, product and security stakeholders.
• Improve metadata, lineage, documentation and data consumer experience.

Skills

Data modelling

Data governance

Mentoring

Strong communication

Python

Advanced SQL

CI/CD practices

GCP familiarity

Tools

Databricks

Apache Spark

Delta Lake

Unity Catalog

Structured Streaming

BigQuery

dbt

Python

SQL

GitHub Actions

Job description

Huxley is seeking a Senior Data Engineer to join a data-mesh driven team building a foundational customer data product. You will design and operate batch, streaming and near-real-time pipelines consolidating sales, payments, orders and gift-card data.

The role requires hands-on Databricks, Spark, Delta Lake and Unity Catalog expertise, strong Python/SQL skills and experience with governance, data-modeling and secure data delivery to analysts and business teams.