Senior Databricks Data Engineer

Version1

Dublin

Remote

EUR 120,000 - 150,000

Full time

14 days+

Get a reply from this employer

Your resume and cover letter, tailored to exactly what they’re hiring for.

Get past ATS filters

Benefits offered by this job

Profit sharing
Career development
Flexible/remote working
Private healthcare
Life insurance
Wellbeing programs
Educational sponsorship
Annual Excellence Award

Job summary

Version 1 in Ireland is seeking an experienced Senior Data Engineer to design and build modern data platforms and AI workflows using Databricks and Azure in an enterprise setting.

You will develop RAG pipelines, MLops, and lakehouse architectures with Bronze-Silver-Gold layers, collaborating across teams to scale AI capabilities.

Qualifications

  • Experience designing and building production-grade data platforms and AI workflows.
  • Hands-on with Retrieval Augmented Generation (RAG) pipelines and AI agent frameworks.
  • Strong Python and SQL programming and hands-on Spark experience.
  • Experience with Databricks Lakehouse architecture and Delta Lake.
  • Familiarity with MLflow for model training/inference pipelines.
  • Knowledge of Azure data services and DevOps practices.

Responsibilities

  • Design and build scalable ETL/ELT and AI pipelines using Databricks, PySpark, Delta Lake and SQL.
  • Develop and maintain Lakehouse architectures with Bronze, Silver and Gold layers.
  • Build and optimise Delta Live Tables, Databricks Workflows and job orchestration processes.
  • Develop and deploy production-grade Generative AI and agent-based solutions.
  • Create and maintain RAG pipelines and AI agent frameworks.
  • Support MLOps activities, including model training and inference pipelines using MLflow.
  • Implement data quality monitoring, governance controls and metadata management via Unity Catalog.
  • Develop event-driven integrations using Azure cloud services.
  • Collaborate with teams to create feature stores and AI training datasets.
  • Integrate Databricks with Azure ADLS Gen2, Data Factory, Key Vault and DevOps.
  • Contribute to data engineering best practices and centers of excellence.
  • Mentor and support engineering teams in AI technologies.

Skills

Senior Data Engineer
AI Engineer
Multi-agent AI workflows
RAG pipelines
Python
SQL
English communication

Tools

Databricks
PySpark
Delta Lake
Unity Catalog
MLflow
Azure Databricks
Azure Service Bus
Azure Functions
Git / CI/CD
Azure Data Factory
Azure Data Lake Storage Gen2

Job description

Overview

We are seeking an experienced Senior Data Engineer to join a high-impact team delivering innovative data and AI solutions within a complex enterprise environment. This role will be responsible for designing and implementing modern data platforms, AI workflows, and Generative AI solutions using Databricks and Azure technologies.

The successful candidate will combine strong data engineering expertise with hands-on experience building production-grade AI solutions, including Retrieval Augmented Generation (RAG) pipelines, multi-agent workflows, and MLOps capabilities.

Key Responsibilities
  • Design and build scalable ETL/ELT and AI pipelines using Databricks, PySpark, Delta Lake and SQL.
  • Develop and maintain Lakehouse architectures using Bronze, Silver and Gold data layers.
  • Build and optimise Delta Live Tables, Databricks Workflows and job orchestration processes.
  • Develop and deploy production-grade Generative AI and agent-based solutions.
  • Create and maintain RAG pipelines and AI agent frameworks.
  • Support MLOps activities, including model training and inference pipelines using MLflow.
  • Implement data quality monitoring, governance controls and metadata management through Unity Catalog.
  • Develop event-driven integrations using Azure cloud services.
  • Collaborate with cross-functional teams to create feature stores and AI training datasets.
  • Integrate Databricks with wider Azure services including ADLS Gen2, Azure Data Factory, Azure Key Vault and Azure DevOps.
  • Contribute to the development of data engineering best practices and centre of excellence initiatives.
  • Mentor and support engineering teams in AI technologies and capabilities.
Essential Skills & Experience
  • Strong experience as a Data Engineer, AI Engineer, Machine Learning Engineer, or similar role.
  • Hands-on experience building and deploying production-grade: Multi-agent AI workflows
  • Retrieval Augmented Generation (RAG) solutions
  • AI agent frameworks
  • Data pipeline solutions within Databricks
  • Strong programming skills in Python and SQL.
  • Experience with Apache Spark and distributed data processing.
  • Experience implementing Generative AI solutions within Databricks Workflows and Jobs.
  • Strong knowledge of Databricks Lakehouse architecture.
  • Hands-on experience with: Delta Lake
  • Unity Catalog
  • Databricks Workflows
  • MLflow
  • Experience implementing event-driven integrations using: Azure Service Bus (essential)
  • Azure Functions (essential)
  • Azure Databricks (essential)
  • Understanding of data modelling and Medallion/Lakehouse architecture.
  • Experience with Git-based version control and CI/CD practices.
  • Strong understanding of AI/ML pipelines, including data preparation, feature engineering and model deployment.
  • Excellent English communication skills.
Desirable Skills
  • Experience with PyTorch and/or TensorFlow.
  • Experience with Event Hub and Logic Apps.
  • Experience designing enterprise-scale AI platforms.
  • Experience mentoring engineering teams or leading technical initiatives.
  • Exposure to financial services or highly regulated environments.
  • Experience building advanced LLMOps capabilities including prompt tracking, trace logging and regression benchmarking.
Technology Stack
  • Databricks
  • PySpark
  • Delta Lake
  • Unity Catalog
  • MLflow
  • Python
  • SQL
  • Apache Spark
  • Azure Databricks
  • Azure Service Bus
  • Azure Functions
  • Azure Data Factory
  • Azure Data Lake Storage Gen2
  • Azure Key Vault
  • Azure DevOps
  • Git
  • PyTorch / TensorFlow
What We're Looking For

We are looking for an engineer who can bridge the gap between modern data engineering and AI innovation. You should be comfortable delivering production-ready solutions, collaborating across teams, and enabling organisations to scale their AI capabilities through robust engineering practices.

Why Version 1?
  • Share in our success with our Quarterly Performance-Related Profit Share Scheme, where employees collectively benefit from a share of our company's profits
  • Strong Career Progression & mentorship coaching through our Strength in Balance & Leadership schemes with a dedicated quarterly Pathways Career Development programme
  • Flexible/remote working, Version 1 is tremendously understanding of life events and people's individual circumstances and offer flexibility to help achieve a healthy work life balance
  • Financial Wellbeing initiatives including; Pension, Private Healthcare Cover, Life Assurance, Financial advice and an Employee Discount scheme
  • Employee Wellbeing schemes including Gym Discounts, Bike to Work, Fitness classes, Mindfulness Workshops, Employee Assistance Programme and much more. Generous holiday allowance, enhanced maternity/paternity leave, marriage/civil partnership leave and special leave policies
  • Educational assistance, incentivised certifications, and accreditations, including AWS, Microsoft, Oracle, and Red Hat
  • Reward schemes including Version 1's Annual Excellence Award
Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Senior Databricks Data Engineer
Senior Databricks Data Engineer

Version 1 • Roebuck

Remote
EUR 95,000 - 135,000
Profit Share Scheme
Career Development & Mentorship
Flexible/remote working
+1
Senior AI Data Engineer
Senior AI Data Engineer

Version 1 • Dublin

Hybrid
EUR 120,000 - 180,000
Profit sharing
Career development
Flexible remote working
+5
Senior Data Engineer (Microsoft Fabric)
Senior Data Engineer (Microsoft Fabric)

Version1 • Dublin

Hybrid
EUR 90,000 - 130,000
Profit sharing scheme
Career progression and mentorship
Flexible/remote working
+2
Azure DevOps Consultant
Azure DevOps Consultant

Version1 • Dublin

Hybrid
EUR 70,000 - 110,000
Quarterly Profit-Share Scheme
Career progression programmes
Flexible/remote working
Applied AI Architect
Applied AI Architect

Version 1 • Dublin

Remote
EUR 150,000 - 210,000
Profit sharing programme
Career development and mentorship
Flexible/remote working
Applied AI Consultant
Applied AI Consultant

Version 1 • Dublin

Remote
EUR 120,000 - 190,000
Profit Share Scheme
Career Progression & Mentorship
Flexible/Remote Working
+6
Ai Engineer
Ai Engineer

Version 1 • Dublin

Remote
GBP 90,000 - 120,000
Quarterly profit share
Career development programs
Flexible/remote working
+3
Azure Integration Developer
Azure Integration Developer

Version1 • Dublin

Hybrid
EUR 70,000 - 110,000
Flexible/remote working
Pension
Private Healthcare
+4
Power Platform Developer
Power Platform Developer

Version1 • Dublin

Hybrid
EUR 80,000 - 110,000
Flexible/remote working
Pension
Private healthcare
+1
Service Delivery Manager
Service Delivery Manager

Version1 • Dublin

Hybrid
EUR 70,000 - 110,000
Pension
Private healthcare
Life assurance
+2
Top locations