Data lead

GCS Recruitment; Sint-Joost-ten-Node; Sur place; EUR 90 000 - 140 000; Plein temps

Résumé du poste

GCS Recruitment is seeking a Databricks Data Lead to provide technical leadership in the design, development, and delivery of modern cloud-based data platforms using Databricks. You will establish data engineering standards and lead technical delivery to ensure scalable, secure, and reliable data solutions aligned with business goals.

The Databricks Data Lead will collaborate with data architects, engineers, analysts, scientists, and cloud teams to enable enterprise data and analytics

Qualifications

• Significant experience in data engineering, data architecture, or cloud data platforms.
• Strong hands-on experience with Databricks and the Lakehouse architecture.
• Strong knowledge of Apache Spark, PySpark, and SQL.
• Experience designing enterprise-scale data pipelines.
• Strong understanding of Delta Lake and modern lakehouse architectures.
• Experience with cloud platforms such as Azure, AWS, or GCP.
• Strong understanding of data warehousing, data lakes, data modelling, and data integration.
• Experience developing both batch and streaming data solutions.
• Knowledge of data governance, security, and access-control principles.
• Experience with CI/CD, automated testing, and DevOps practices.
• Experience with data orchestration tools such as Databricks Workflows, Apache Airflow, Azure Data Factory.

Responsabilités

• Lead the design and implementation of scalable enterprise data platforms using Databricks.
• Provide technical leadership across data engineering, data integration, data transformation, and analytics initiatives.
• Define and implement modern data architecture patterns using the Databricks Lakehouse architecture.
• Design and implement data pipelines using Spark, PySpark, SQL, and Databricks.
• Establish standards and best practices for data ingestion, transformation, processing, storage, and consumption.
• Lead the development of batch and real-time/streaming data pipelines.
• Design and implement data lake and lakehouse solutions using Delta Lake.
• Define approaches for data modelling, data partitioning, optimisation, and storage management.
• Lead implementation of data governance, security, access control, data quality, lineage, and metadata management.
• Work with architects to establish integration patterns between Databricks and enterprise data sources.

Connaissances

Databricks

Lakehouse

Spark

PySpark

SQL

Cloud platforms

CI/CD

DevOps

Data governance

Leadership

Data modelling

Outils

Databricks

Apache Airflow

Azure Data Factory

Terraform

dbt

Description du poste

Databricks Data Lead

Role Overview

We are seeking an experienced Databricks Data Lead to provide technical leadership in the design, development, and delivery of modern cloud-based data platforms using Databricks. The role will be responsible for establishing data engineering standards, leading technical delivery, and ensuring that data solutions are scalable, secure, reliable, and aligned with business objectives.

The Databricks Data Lead will work closely with data architects, data engineers, analysts, data scientists, business stakeholders, and cloud engineering teams to deliver enterprise data and analytics capabilities.

Key Responsibilities

• Lead the design and implementation of scalable enterprise data platforms using Databricks.
• Provide technical leadership across data engineering, data integration, data transformation, and analytics initiatives.
• Define and implement modern data architecture patterns using the Databricks Lakehouse architecture.
• Design and implement data pipelines using Apache Spark, PySpark, SQL, and Databricks.
• Establish standards and best practices for data ingestion, transformation, processing, storage, and consumption.
• Lead the development of batch and real-time/streaming data pipelines.
• Design and implement data lake and lakehouse solutions using technologies such as Delta Lake.
• Define approaches for data modelling, data partitioning, optimisation, and storage management.
• Lead the implementation of data governance, security, access control, data quality, lineage, and metadata management.
• Work with architects and engineering teams to establish appropriate integration patterns between Databricks and enterprise data sources and platforms.
• Drive performance optimisation and cost management across Databricks workloads.
• Establish development standards covering coding, testing, deployment, monitoring, and operational support.
• Implement and promote CI/CD and DevOps practices for data engineering solutions.
• Support Infrastructure as Code and automated deployment approaches where appropriate.
• Lead technical design reviews and provide guidance on complex data engineering challenges.
• Mentor and support data engineers, helping establish consistent engineering practices and technical standards.
• Work with delivery managers and product owners to translate business requirements into technical solutions.
• Estimate technical effort, identify dependencies and risks, and contribute to delivery planning.
• Troubleshoot and resolve complex data pipeline, performance, reliability, and integration issues.
• Produce and maintain technical documentation, architecture diagrams, design specifications, and engineering standards.
• Evaluate new Databricks capabilities and emerging data technologies and recommend opportunities for adoption.

Required Skills and Experience

• Significant experience in data engineering, data architecture, or cloud data platforms.
• Strong hands‑on experience with Databricks and the Lakehouse architecture.
• Strong knowledge of Apache Spark, PySpark, and SQL.
• Experience designing and implementing enterprise‑scale data pipelines.
• Strong understanding of Delta Lake and modern data lake/lakehouse architectures.
• Experience working with cloud platforms such as Azure, AWS, or Google Cloud Platform.
• Strong understanding of data warehousing, data lakes, data modelling, and data integration.
• Experience developing both batch and streaming data solutions.
• Strong understanding of data quality, governance, security, and access‑control principles.
• Experience with source control, CI/CD, automated testing, and DevOps practices.
• Experience with data orchestration technologies such as Databricks Workflows, Apache Airflow, Azure Data Factory, or equivalent.
• Experience integrating data from relational databases, APIs, files, event streams, and enterprise applications.
• Strong programming experience in Python and/or Scala.
• Experience leading technical teams or providing technical leadership to data engineering teams.
• Strong problem‑solving, communication, and stakeholder‑management skills.

Desirable Skills

• Experience with Unity Catalog and Databricks governance capabilities.
• Experience with Delta Live Tables / Lakeflow Declarative Pipelines.
• Experience with Databricks SQL and SQL Warehouses.
• Experience with Structured Streaming and technologies such as Kafka.
• Experience with dbt or other modern data transformation frameworks.
• Experience with Terraform or other Infrastructure as Code technologies.
• Experience implementing data mesh or domain‑oriented data architectures.
• Experience with data cataloguing, lineage, and data governance platforms.
• Experience with cloud‑native services across Azure, AWS, or GCP.
• Experience with machine learning, AI, or advanced analytics platforms.
• Databricks or relevant cloud certifications.

Leadership Responsibilities

• Set technical direction and engineering standards for the data platform.
• Lead and mentor data engineers and provide technical guidance across delivery teams.
• Review technical designs and ensure alignment with enterprise architecture principles.
• Identify opportunities to improve platform scalability, reliability, performance, and cost efficiency.
• Promote engineering best practices, automation, reusable components, and standardised patterns.
• Act as a technical point of contact between engineering teams, architecture teams, and business stakeholders.
• Support recruitment, technical assessments, capability development, and knowledge sharing where required.

Key Competencies

• Databricks & Lakehouse Architecture
• Apache Spark & PySpark
• Data Engineering
• Data Architecture & Modelling
• Delta Lake
• Cloud Data Platforms
• Data Governance & Security
• Data Pipeline Design
• Performance & Cost Optimisation
• CI/CD & DevOps
• Technical Leadership
• Stakeholder Management
• Mentoring & Team Development

Typical Experience Level

7-12+ years of experience in data engineering, data architecture, analytics engineering, or related disciplines, including significant hands‑on experience with Databricks and cloud‑based data platforms.

A strong candidate should be capable of operating at both strategic and hands‑on technical levels, providing architectural direction while remaining close enough to the engineering detail to guide implementation and resolve complex technical challenges.

GCS is acting as an Employment Business in relation to this vacancy.