Senior / Lead Data Engineer - DataBricks

Cpl; Dublin; Vor Ort; EUR 90.000 - 130.000; Vollzeit

Zusammenfassung

CPL is seeking a Data Engineering Lead to own the design and delivery of scalable data pipelines and lakehouse solutions on Databricks. You will drive architectural decisions, mentor engineers, and ensure governance and quality across platforms.

The role requires 5+ years in data engineering with 3+ years on Databricks, strong SQL/PySpark, and a focus on reliability, cost, and performance. Commutable to Dublin, full-time, fixed-term with view to permanency.

Qualifikationen

• 5+ years in data engineering
• 3+ years hands-on on Databricks
• Strong SQL and PySpark/Spark SQL expertise
• Experience with Delta Lake and Unity Catalog
• Familiarity with Databricks tooling and CI/CD
• Must have the relevant Visa for a full time position and live within a commutable distance to Dublin

Aufgaben

• Pipeline Engineering — Design and own scalable ETL/ELT pipelines (PySpark, Spark SQL, Delta Live Tables) following medallion architecture.
• Data Modelling & Lakehouse — Govern Delta Lake models and modelling standards across Bronze/Silver/Gold layers.
• Platform & Architecture — Own Databricks platform architecture and tooling decisions (Delta Lake, Unity Catalog, Databricks SQL, structured streaming).
• Reliability & Observability — Define SLAs and implement observability (lineage, monitoring, metrics).
• Leadership & Collaboration — Act as technical lead for the data engineering function; mentor engineers; lead design reviews and architecture discussions.
• Governance & Auditability — Ensure traceable, well-governed data processes via Unity Catalog; maintain documentation and compliance.
• Partner Management — Oversee external delivery partners, ensuring quality and knowledge transfer.

Kenntnisse

Databricks

SQL

PySpark

Delta Lake

Unity Catalog

CI/CD

Leadership

Stakeholder comms

Medallion architecture

Ausbildung

Bachelor's degree in CS/Data Engineering

Tools

Databricks Workflows

Jobbeschreibung

Owns the design and optimisation of scalable data pipelines and lakehouse solutions on Databricks, enabling analytics and data-driven decision-making across the organisation. Acts as a technical leader — setting standards, mentoring engineers, and shaping the platform's evolution.

This is a fixed term contract initially with a view to permanency.

Core Competencies

• Technical leadership and architectural decision-making on Databricks
• Delivery of business value through scalable lakehouse solutions
• Strong stakeholder communication and cross-team collaboration
• Continuous improvement and engineering excellence

Key Responsibilities

• Pipeline Engineering — Design and own scalable ETL/ELT pipelines (PySpark, Spark SQL, Delta Live Tables) following medallion architecture, with strong reliability, monitoring, and cost/performance optimisation.
• Data Modelling & Lakehouse — Govern Delta Lake models and modelling standards across Bronze/Silver/Gold layers, ensuring data quality, integrity, and lifecycle management.
• Platform & Architecture — Own Databricks platform architecture and tooling decisions (Delta Lake, Unity Catalog, Databricks SQL, structured streaming), aligned with enterprise security and governance.
• Reliability & Observability — Define SLAs and implement observability (lineage, monitoring, metrics); resolve performance bottlenecks and manage cost.
• Leadership & Collaboration — Act as technical lead for the data engineering function; mentor engineers; lead design reviews and architecture discussions.
• Governance & Auditability — Ensure traceable, well-governed data processes via Unity Catalog; maintain documentation and support compliance.
• Partner Management — Oversee external delivery partners, ensuring quality, standards adherence, and knowledge transfer.

Experience & Skills

• 5+ years in data engineering, with 3+ years hands-on on Databricks
• Strong SQL and PySpark/Spark SQL expertise
• Proven experience with Delta Lake, Unity Catalog, Databricks Workflows, and Databricks SQL
• Solid grasp of medallion architecture, ETL/ELT design, and Spark performance tuning
• CI/CD experience (Databricks Asset Bundles, Git/Repos)
• Familiarity with structured streaming, Auto Loader, and batch/near-real-time processing
• Databricks certification (Data Engineer Associate/Professional) desirable
• Must have the relevant Visa for a full time position and live within a commutable distance to Dublin

Education

Degree in Computer Science, Data Engineering, or related field