Senior / Lead Data Engineer - DataBricks

Cpl; Dublin; On-site; EUR 90,000 - 130,000; Full time

Job summary

CPL is seeking a technical leader to own the design and optimisation of scalable data pipelines and lakehouse solutions on Databricks, enabling analytics and data‑driven decision making across CPL.

This is a fixed term contract initially with a view to permanency. Based in Ireland, you will mentor engineers, set standards, and shape the platform's evolution while collaborating with stakeholders. Must be based within a commutable distance to Dublin.

Qualifications

• 5+ years in data engineering with 3+ years on Databricks.
• Strong SQL and PySpark/Spark SQL expertise.
• Experience with Delta Lake, Unity Catalog, Databricks Workflows, and Databricks SQL.
• Must be based within a commutable distance to Dublin and eligible to work in Ireland.

Responsibilities

• Design and own scalable ETL/ELT pipelines using PySpark and Delta Live Tables.
• Govern Delta Lake models across Bronze/Silver/Gold layers for quality and lifecycle management.
• Own Databricks platform architecture and tooling decisions (Delta Lake, Unity Catalog, Databricks SQL).
• Define SLAs and implement observability, monitoring, and cost/performance optimisation.
• Mentor engineers and lead design reviews and architecture discussions.
• Ensure traceability and governance via Unity Catalog and maintain documentation.
• Oversee external delivery partners to ensure quality and knowledge transfer.

Skills

Databricks

SQL

PySpark

Delta Lake

Unity Catalog

CI/CD

Data modeling

Platform architecture

Education

Degree in Computer Science

Tools

Delta Live Tables

Auto Loader

Structured Streaming

Git/Repos

Databricks Workflows

Job description

Owns the design and optimisation of scalable data pipelines and lakehouse solutions on Databricks, enabling analytics and data-driven decision-making across the organisation. Acts as a technical leader — setting standards, mentoring engineers, and shaping the platform's evolution. This is a fixed term contract initially with a view to permanency.

Core Competencies

• Technical leadership and architectural decision-making on Databricks
• Delivery of business value through scalable lakehouse solutions
• Strong stakeholder communication and cross-team collaboration
• Continuous improvement and engineering excellence

Key Responsibilities

• Pipeline Engineering — Design and own scalable ETL/ELT pipelines (PySpark, Spark SQL, Delta Live Tables) following medallion architecture, with strong reliability, monitoring, and cost/performance optimisation.
• Data Modelling & Lakehouse — Govern Delta Lake models and modelling standards across Bronze/Silver/Gold layers, ensuring data quality, integrity, and lifecycle management.
• Platform & Architecture — Own Databricks platform architecture and tooling decisions (Delta Lake, Unity Catalog, Databricks SQL, structured streaming), aligned with enterprise security and governance.
• Reliability & Observability — Define SLAs and implement observability (lineage, monitoring, metrics); resolve performance bottlenecks and manage cost.
• Leadership & Collaboration — Act as technical lead for the data engineering function; mentor engineers; lead design reviews and architecture discussions.
• Governance & Auditability — Ensure traceable, well-governed data processes via Unity Catalog; maintain documentation and support compliance.
• Partner Management — Oversee external delivery partners, ensuring quality, standards adherence, and knowledge transfer.

Experience & Skills

• 5+ years in data engineering, with 3+ years hands-on on Databricks
• Strong SQL and PySpark/Spark SQL expertise
• Proven experience with Delta Lake, Unity Catalog, Databricks Workflows, and Databricks SQL
• Solid grasp of medallion architecture, ETL/ELT design, and Spark performance tuning
• CI/CD experience (Databricks Asset Bundles, Git/Repos)
• Familiarity with structured streaming, Auto Loader, and batch/near-real-time processing
• Databricks certification (Data Engineer Associate/Professional) desirable
• Must be based within a commutable distance to Dublin
• Must have the relevant work status to work in Ireland.

Education

Degree in Computer Science, Data Engineering, or related field