Databricks Data Platform Lead: Lakehouse & Pipelines

Cpl; Dublin; On-site; EUR 90,000 - 130,000; Full time

Job summary

CPL is seeking a technical leader to own the design and optimisation of scalable data pipelines and lakehouse solutions on Databricks, enabling analytics and data‑driven decision making across CPL.

This is a fixed term contract initially with a view to permanency. Based in Ireland, you will mentor engineers, set standards, and shape the platform's evolution while collaborating with stakeholders. Must be based within a commutable distance to Dublin.

Qualifications

• 5+ years in data engineering with 3+ years on Databricks.
• Strong SQL and PySpark/Spark SQL expertise.
• Experience with Delta Lake, Unity Catalog, Databricks Workflows, and Databricks SQL.
• Must be based within a commutable distance to Dublin and eligible to work in Ireland.

Responsibilities

• Design and own scalable ETL/ELT pipelines using PySpark and Delta Live Tables.
• Govern Delta Lake models across Bronze/Silver/Gold layers for quality and lifecycle management.
• Own Databricks platform architecture and tooling decisions (Delta Lake, Unity Catalog, Databricks SQL).
• Define SLAs and implement observability, monitoring, and cost/performance optimisation.
• Mentor engineers and lead design reviews and architecture discussions.
• Ensure traceability and governance via Unity Catalog and maintain documentation.
• Oversee external delivery partners to ensure quality and knowledge transfer.

Skills

Databricks

SQL

PySpark

Delta Lake

Unity Catalog

CI/CD

Data modeling

Platform architecture

Education

Degree in Computer Science

Tools

Delta Live Tables

Auto Loader

Structured Streaming

Git/Repos

Databricks Workflows

Job description

CPL is seeking a technical leader to own the design and optimisation of scalable data pipelines and lakehouse solutions on Databricks, enabling analytics and data‑driven decision making across CPL.

This is a fixed term contract initially with a view to permanency. Based in Ireland, you will mentor engineers, set standards, and shape the platform's evolution while collaborating with stakeholders. Must be based within a commutable distance to Dublin.