Jr. and Sr. Data Engineer retail

Huxley; Amsterdam; On-site; EUR 90,000 - 135,000; Full time

Benefits offered by this job

null

Job summary

Huxley is seeking a Senior Data Engineer to join a data-mesh driven team building a foundational customer data product. You will design and operate batch, streaming and near-real-time pipelines consolidating sales, payments, orders and gift-card data.

The role requires hands-on Databricks, Spark, Delta Lake and Unity Catalog expertise, strong Python/SQL skills and experience with governance, data-modeling and secure data delivery to analysts and business teams.

Qualifications

• Senior-level experience designing, building and operating production data platforms and data products.
• Hands-on Databricks experience including Spark, Delta Lake, Unity Catalog and Structured Streaming.
• Strong Python and advanced SQL skills.
• Experience with distributed data processing and scalable pipelines (batch and streaming).
• Data-modelling, data-quality validation, monitoring and alerting.
• Knowledge of event-driven architecture, schema evolution and error-handling patterns.
• Experience with Git, CI/CD and modern software engineering practices.
• Knowledge of data security, privacy, metadata and data lineage.

Responsibilities

• Design, build, test and operate batch, streaming and near-real-time data pipelines.
• Ingest and harmonise data from multiple source systems (sales, payments, orders, gift-card).
• Develop transformations and production data models using SQL, Python, BigQuery and dbt.
• Apply Databricks and Spark expertise to scalable processing and governance decisions.
• Implement data-quality validation, monitoring and automated checks.
• Mentor colleagues and collaborate with analysts, product and security stakeholders.
• Improve metadata, lineage, documentation and data consumer experience.

Skills

Data modelling

Data governance

Mentoring

Strong communication

Python

Advanced SQL

CI/CD practices

GCP familiarity

Tools

Databricks

Apache Spark

Delta Lake

Unity Catalog

Structured Streaming

BigQuery

dbt

Python

SQL

GitHub Actions

Job description

Background

We're looking for a Junior and Senior Data Engineer for our retail client to work on a foundational customer data product that consolidates and harmonises sales and payment transactional data from multiple source systems. It covers the sales transaction lifecycle from the initial payment attempt through order creation and handover to the fulfilment system.

The product integrates online and in-store sales, payments and gift-card transactions into a consistent and accessible source of data. It enables analysts and other data consumers to understand customer behaviour, monitor business performance and develop analytical products without spending unnecessary time finding, combining and cleaning data.

Built according to data-mesh principles, the product supports domain-driven ownership and federated governance. The product focuses on data accuracy, consistency, accessibility and usability, providing trusted, ready-for-consumption data through BigQuery. It also supports the wider modernisation journey by providing harmonised sales and payment data to analytical products and business teams.

Assignment

The consultant will join the team as a Senior Data Engineer and contribute to the design, development and operation of a foundational data product that consolidates and harmonises customer sales, payment, order and gift-card transactional data.

The assignment includes building reliable batch, streaming and near-real-time data pipelines; developing reusable data models; improving data quality, monitoring and resilience; and supporting the secure and accessible delivery of trusted data to analysts and other consumers.

As a senior team member, the consultant will help guide architectural decisions and strengthen engineering practices. The consultant will bring practical Databricks and lakehouse expertise, including Apache Spark, Delta Lake, Unity Catalog and Structured Streaming, while working with and supporting the current GCP-based technology stack.

Day-to-Day Tasks

• Design, build, test and operate batch, streaming and near-real-time data pipelines.
• Ingest and harmonise sales, payment, order and gift-card data from multiple source systems.
• Develop transformations and production data models using SQL, Python, BigQuery and dbt.
• Apply Databricks and Spark experience to scalable processing, streaming, governance and lakehouse-related design decisions.
• Implement validation, reconciliation, monitoring and automated data-quality controls.
• Build resilient processing with error handling, recovery mechanisms and dead-letter queues.
• Monitor data freshness, pipeline reliability, performance and agreed service levels.
• Protect sensitive and personally identifiable information through secure data modelling and access controls.
• Contribute to automated testing, CI/CD, code reviews and infrastructure automation.
• Improve metadata, lineage, documentation and the experience of data consumers.
• Mentor colleagues and collaborate with source teams, analysts, product colleagues, privacy specialists and security stakeholders.

Requirements

• Senior-level experience designing, building and operating production data platforms and business-critical data products.
• Strong hands-on Databricks experience, including Apache Spark, Delta Lake, Unity Catalog, Structured Streaming, Databricks Workflows and lakehouse architecture.
• Strong Python and advanced SQL skills.
• Experience with distributed data processing and scalable batch and streaming pipelines.
• Strong data-modelling skills and experience developing reliable, reusable production data models.
• Experience with data-quality validation, reconciliation, monitoring and alerting.
• Knowledge of event-driven architecture, change data capture, schema evolution and resilient error-handling patterns.
• Experience with Git, automated testing, CI/CD and modern software-engineering practices.
• Knowledge of data security, IAM, privacy, PII handling, metadata and data lineage.
• Strong communication skills and the ability to guide architectural decisions, mentor colleagues and collaborate with technical and business stakeholders.

Desired

• Experience with GCP, particularly BigQuery, Dataflow, Pub/Sub, Cloud Run and Cloud Storage.
• Experience with dbt, Apache Beam, Terraform and GitHub Actions.
• Experience working with sales, payments, orders or other high-volume transactional data.
• Understanding of data-mesh principles, domain ownership and federated governance.
• Experience in a product-oriented Agile or DevOps environment.

Top 3 Priorities

• Extensive hands-on Databricks expertise, particularly Apache Spark, Delta Lake, Unity Catalog, Structured Streaming and lakehouse architecture.
• Senior-level experience designing and operating reliable, scalable batch and streaming data pipelines for business-critical transactional data.
• Strong technical leadership, including architectural decision-making, mentoring and clear collaboration with technical and business stakeholders.