Senior Data Engineer

Corecom Consulting; Randstad; On-site; EUR 90,000 - 135,000; Full time

Job summary

Corecom Consulting seeks a Senior Data Engineer to join a team delivering a foundational data product aggregating sales, payments, orders and gift-card data from multiple sources on a GCP-based stack (BigQuery, dbt, Delta Lake).

You will design and operate scalable pipelines, implement data quality controls, and help shape lakehouse governance while collaborating with analysts, product teams and stakeholders.

Qualifications

• Senior-level experience designing and operating production data platforms.
• Hands-on Databricks experience with Spark, Delta Lake, Unity Catalog and structured streaming.
• Strong Python and advanced SQL skills.
• Proven ability to build scalable batch and streaming pipelines.
• Knowledge of data mesh, domain ownership and governance.

Responsibilities

• Design, build and operate scalable batch and streaming data pipelines.
• Ingest and harmonize transactional data from multiple sources.
• Develop transformations and production data models using Python, SQL, BigQuery and dbt.
• Apply Databricks and Spark expertise to lakehouse architecture.
• Improve data quality with validation, testing and monitoring.
• Mentor engineers and contribute to architectural decisions.

Skills

Databricks

Python

SQL

BigQuery

Spark

GCP

CI/CD

Data governance

Data quality

Data mesh

Tools

dbt

Terraform

GitHub Actions

Unity Catalog

Delta Lake

Job description

We are looking for an experienced Senior Data Engineer to join a team developing a foundational data product that brings together sales, payment, order and gift-card transactional data from multiple source systems.

The product creates a consistent, trusted source of data across online and in-store transactions, making it easier for analysts and business teams to understand customer behavior, monitor performance and build analytical solutions.

The platform is built around modern data engineering and data-mesh principles, with a focus on data quality, accessibility, governance and reliability. The existing environment is primarily GCP-based, with BigQuery at its core.

This is an opportunity for a senior engineer who can bring strong hands-on Databricks and lakehouse expertise , helping shape technical decisions and strengthen the way data is processed, governed and delivered.

What you'll be doing

• Design, build and operate scalable batch, streaming and near-real-time data pipelines.
• Ingest, process and harmonize sales, payment, order and gift card data from multiple sources.
• Develop reliable transformations and production data models using Python, SQL, BigQuery and dbt.
• Apply Databricks and Apache Spark expertise to scalable data processing, streaming and lakehouse architecture.
• Work with technologies including Delta Lake, Unity Catalog and Databricks Workflows.
• Improve data quality through validation, reconciliation, automated testing and monitoring.
• Build resilient pipelines with effective error handling, recovery mechanisms and dead-letter queues.
• Monitor pipeline performance, data freshness and reliability.
• Contribute to data security, access controls, metadata management and data lineage.
• Support engineering best practices across CI/CD, Git, code reviews and infrastructure automation.
• Provide technical guidance, contribute to architectural decisions and mentor other engineers.
• Work closely with engineers, analysts, product teams and wider business stakeholders.

What we're looking for

• Strong senior-level experience designing and operating production data platforms and business-critical data products.
• Extensive hands-on experience with Databricks , particularly Apache Spark, Delta Lake, Unity Catalog and structured streaming.
• Strong Python and advanced SQL skills.
• Proven experience building scalable batch and streaming data pipelines.
• Strong understanding of lakehouse architecture and distributed data processing.
• Experience developing reliable data models and implementing data-quality controls.
• Understanding of event-driven architecture, change data capture, schema evolution and resilient processing patterns.
• Experience with Git, automated testing, CI/CD and modern software engineering practices.
• Knowledge of data security, IAM, privacy, PII handling and data governance.
• Ability to take ownership, influence technical decisions and communicate effectively with both technical and business stakeholders.
• GCP experience, particularly BigQuery, Dataflow, Pub/Sub, Cloud Run and Cloud Storage.
• dbt, Apache Beam, Terraform and GitHub Actions.
• Working with high-volume transactional datasets, ideally sales, payments or orders.
• Data mesh principles, domain ownership and federated governance.
• Experience in product-oriented Agile or DevOps environments.

The ideal consultant

We're particularly interested in people who combine deep practical Databricks expertise with the ability to think beyond individual pipelines.

You’ll be comfortable making architectural decisions, improving engineering standards and helping a team build reliable, scalable data products. You should be equally comfortable getting hands‑on with the code as discussing technical trade-offs with colleagues and stakeholders.