Senior Data Engineer - Databricks Lakehouse & AI Pipelines

Jobtailor; Dublin; On-site; EUR 90,000 - 140,000; Full time

Job summary

Jobtailor is seeking a senior Data Engineer Leader in Dublin to act as the technical authority for Databricks Data Engineering and define the data architecture for analytics data products.

You will design conceptual, logical, and physical data models, set engineering standards for AI-assisted pipelines, and oversee large-scale data orchestration across medallion layers in Databricks.

Qualifications

• Must have strong leadership and mentoring experience.
• Extensive data engineering experience in cloud environments.
• Proven track record delivering large-scale data initiatives.
• Experience in regulated industries such as Insurance is a plus.

Responsibilities

• Act as the leading technical authority for Databricks Data Engineering.
• Define and own the Data Architecture for analytics data products.
• Design data models aligned to standardised models.
• Establish engineering standards and best practices for AI-assisted pipelines.

Skills

Leadership / Mentoring

Cloud data engineering

Data pipelines design

Regulated environments

Education

BSc/MSc in CS or Engineering

Data Management certifications (DAMA)

Tools

Databricks Lakehouse

AWS Cloud

Job description

• Act as the leading technical authority and Subject Matter Expert for Databricks Data Engineering, representing the Engineering team to stakeholders.
• Define and own the Data Architecture for the creation of speciality insurance analytics data products.
• Design Conceptual, Logical, and Physical Data Models for the business area, aligned to broader/standardised data models.
• Establish engineering standards, patterns, and best practices for AI-assisted pipeline development, orchestration, processing, storage, and observability.
• Design, oversee, and contribute to the development of large-scale, fault-tolerant data pipelines and orchestration jobs across medallion layers in Databricks.
• Ensure pipelines are designed to meet SLAs for reliability, latency, cost, and data quality.
• Implement processes for the management of metadata, lineage, and observability in the delivery of high-quality and dependable Data Products.
• Partner with security teams on data privacy and compliance, implementing data governance with particular focus on managing sensitive, business critical data.
• Mentor senior and mid-level engineers and drive Engineering Excellence across AI Assisted Development, Code quality, Testing, and Documentation.
• Lead design reviews and architecture discussions, encouraging collaboration and innovation to promote a culture of continuous improvement.

Requirements

• Databricks and AWS Certifications
• BSc/MSc degree in Computer Science, Engineering or a related subject
• Data Management certifications (e.g. DAMA) an advantage
• Extensive experience in Data Engineering roles
• Proven experience leading or mentoring engineering teams
• Experience delivering large-scale cloud-based data initiatives
• Previous experience working within regulated environments such as Insurance highly desirable
• Experience with Databricks Lakehouse architecture preferred

Core Competencies

Demonstrates expertise in Databricks Data Engineering and Data Architecture, with a strong focus on designing and implementing large-scale data pipelines and ensuring data quality and compliance. Proven ability to mentor engineering teams and establish best practices for AI-assisted development.

Highest-signal resume keywords

• Databricks Data Engineering
• Data Architecture
• Large-Scale Data Pipelines
• AWS Certifications
• Data Management Certifications

ATS Optimization Keywords

Hard Skills

• Data Modeling
• Data Pipeline Development
• Data Governance
• Metadata Management
• Data Quality Assurance

Soft Skills

• Mentoring
• Collaboration
• Continuous Improvement

Certifications & Qualifications

• Databricks Certification
• AWS Certification
• Data Management Certification

Industry Keywords

• Speciality Insurance
• Regulated Environments
• Data Privacy
• Compliance

Tools & Technologies

• Databricks Lakehouse
• AI-Assisted Development
• Cloud-Based Data Solutions