Senior Data Engineer – Databricks Lakehouse Platform

ONE-Consultants; Breda; On-site; EUR 90,000 - 130,000; Full time

Job summary

ONE-Consultants is seeking a Senior Data Engineer to design, build, and optimize modern data platforms using the Databricks Lakehouse Platform. The candidate will lead data platform architecture and implement scalable pipelines across Bronze, Silver, and Gold layers with Medallion Architecture principles.

The role requires 10+ years of data engineering experience and deep expertise in Lakehouse, Unity Catalog, and cloud-native solutions.

Qualifications

• 10+ years of experience in Data Engineering, Data Warehousing, or Big Data platforms.
• 5+ years of hands-on experience with Databricks in enterprise environments.
• Strong expertise in Lakehouse, Medallion Architecture, Unity Catalog.
• Experience with AWS, Azure, or GCP.
• Programming in Python, SQL, PySpark.
• Knowledge of CI/CD and IaC (Terraform).
• Experience with data governance and data quality frameworks.
• Experience in designing cloud-native data solutions.

Responsibilities

• Design and implement scalable, high-performance data platforms using the Databricks Lakehouse architecture.
• Lead the development of enterprise data solutions leveraging Bronze, Silver, and Gold layers following Medallion Architecture principles.
• Build and maintain batch and streaming data pipelines for large-scale data processing.
• Develop reusable data frameworks, accelerators, and engineering best practices.
• Optimize data storage, processing, and query performance across cloud environments.
• Design, develop, and maintain Databricks workflows using PySpark, SQL, and Python.
• Implement and manage Delta Live Tables (DLT) pipelines.
• Utilize Unity Catalog for centralized governance, metadata management, access controls, and lineage tracking.
• Support advanced analytics, machine learning, and AI workloads on the Databricks platform.
• Monitor and troubleshoot production data pipelines and platform performance.
• Establish and enforce enterprise data governance standards and policies.
• Implement data quality frameworks, validation rules, reconciliation processes, and monitoring solutions.
• Ensure compliance with security, privacy, regulatory, and audit requirements.
• Drive metadata management, lineage, cataloging, and data stewardship initiatives.
• Collaborate with business and governance teams to improve trust and usability of enterprise data assets.
• Design and deploy cloud-native data solutions on Azure, AWS, or GCP.
• Implement scalable and secure cloud architectures aligned with enterprise standards.
• Optimize cloud resource utilization, performance, and cost management.
• Integrate cloud-native services with Databricks-based solutions.
• Automate deployment, testing, monitoring, and operational processes.
• Apply Infrastructure as Code (IaC) practices using tools such as Terraform or similar.
• Promote engineering excellence through code reviews, testing strategies, and release management processes.
• Provide technical leadership and mentorship to junior and mid-level engineers.
• Collaborate with architects, analysts, data scientists, product owners, and business stakeholders.
• Contribute to platform roadmaps, architectural decisions, and technology evaluations.
• Lead technical discussions and establish best practices across the data engineering organization.

Skills

Databricks

Python

SQL

PySpark

Delta Live Tables

Unity Catalog

Lakehouse Architecture

Medallion Architecture

Data Governance

Cloud platforms

Education

Bachelor's or Master's degree in CS/IS/Engineering

Tools

Terraform

CI/CD tooling

Job description

This role is for professionals seeking a long-term career with us; subcontractor applications will not be considered.

We are seeking an experiencedSenior Data Engineerto design, build, and optimize modern

data platforms using theDatabricks Lakehouse Platform. The ideal candidate will have10+

years of data engineering experience

years of data engineering experience

and deep expertise inLakehouse

Architecture,Medallion Architecture, cloud-native data solutions, and enterprise-scale data

governance practices.

Key Responsibilities

Data Platform Architecture & Engineering

• Design and implement scalable, high-performance data platforms using the Databricks Lakehouse architecture.
• Lead the development of enterprise data solutions leveraging Bronze, Silver, and Gold layers following Medallion Architecture principles.
• Build and maintain batch and streaming data pipelines for large-scale data processing.
• Develop reusable data frameworks, accelerators, and engineering best practices.
• Optimize data storage, processing, and query performance across cloud environments.

Databricks Development

• Design, develop, and maintain Databricks workflows using PySpark, SQL, and Python.
• Implement and manage Delta Live Tables (DLT) pipelines.
• Utilize Unity Catalog for centralized governance, metadata management, access controls, and lineage tracking.
• Support advanced analytics, machine learning, and AI workloads on the Databricks platform.
• Monitor and troubleshoot production data pipelines and platform performance.

Data Governance & Quality

• Establish and enforce enterprise data governance standards and policies.
• Implement data quality frameworks, validation rules, reconciliation processes, and monitoring solutions.
• Ensure compliance with security, privacy, regulatory, and audit requirements.
• Drive metadata management, lineage, cataloging, and data stewardship initiatives.
• Collaborate with business and governance teams to improve trust and usability of enterprise data assets.
• Design and deploy cloud-native data solutions on Azure, AWS, or GCP.
• Implement scalable and secure cloud architectures aligned with enterprise standards.
• Optimize cloud resource utilization, performance, and cost management.
• Integrate cloud-native services with Databricks-based solutions.

DevOps & Automation

• Automate deployment, testing, monitoring, and operational processes.
• Apply Infrastructure as Code (IaC) practices using tools such as Terraform or similar.
• Promote engineering excellence through code reviews, testing strategies, and release management processes.
• Provide technical leadership and mentorship to junior and mid-level engineers.
• Collaborate with architects, analysts, data scientists, product owners, and business stakeholders.
• Contribute to platform roadmaps, architectural decisions, and technology evaluations.
• Lead technical discussions and establish best practices across the data engineering organization.

Required Qualifications

• Bachelor's or Master's degree in Computer Science, Information Systems, Engineering, or a related field.
• 10+ years of experiencein Data Engineering, Data Warehousing, or Big Data platforms.
• 5+ years of hands-on experience with Databricksin enterprise environments.
• Strong expertise in:
• Lakehouse Architecture
• Medallion Architecture
• Unity Catalog
• Data Governance and Data Quality frameworks
• Data Modeling and Data Warehousing concepts
• Extensive experience with at least one major cloud platform:
• Amazon Web Services (AWS)
• Advanced programming experience with:
• Python
• SQL
• PySpark
• Strong understanding of CI/CD methodologies and DevOps practices.
• Experience implementing Infrastructure as Code (Terraform).
• Expertise in performance tuning, optimization, and troubleshooting of large-scale data pipelines.
• Strong understanding of security, access controls, and compliance requirements in cloud environments.

Preferred Qualifications

• Databricks certifications such as 'Databricks Certified Data Engineer Professional'
• Cloud certifications (Azure, AWS, or GCP).
• Experience with real-time streaming technologies and event-driven architectures.
• Experience supporting AI/ML and Generative AI data platforms.
• Familiarity with data observability tools and modern data quality frameworks.
• Experience implementing enterprise metadata management and data lineage solutions.

Hiring Process

We aim to complete the interview process in October-November. The targeted onboarding is January 2027 or earlier.