Principal Data Engineer

Vermelo RPO; Manchester; Hybrid; GBP 90,000 - 130,000; Full time

Benefits offered by this job

Annual leave 25 days

Yearly bonus

Private Medical Insurance

Life Assurance 4x annual salary

Hybrid Working

Modern, vibrant offices

Job summary

Markerstudy Group Services Limited is seeking a Principal Data Engineer to act as the senior technical authority for the Databricks Lakehouse platform. You will define engineering standards, architecture, and best practices while leading the design and delivery of scalable, secure data products using Databricks, Delta Lake, Unity Catalog and Azure Data Services.

The role requires strong leadership across squads, governance, and resilience, with a focus on end-to-end data product delivery and

Qualifications

• Extensive Data Engineering experience within enterprise-scale data environments.
• Hands-on experience delivering solutions using Databricks.
• Experience designing and implementing Lakehouse architectures.
• Strong commercial experience using PySpark, Python, SQL and Delta Lake.
• Experience optimising Databricks workloads for performance, scalability and cost efficiency.
• Experience delivering solutions in insurance, financial services or regulated industries.
• Ability to translate business needs into technical requirements.
• Good project management skills; experience managing projects with multiple senior stakeholders.
• Experience managing and mentoring engineering teams.

Responsibilities

• Being the primary technical contact on data for new projects and designing data solutions.
• Design scalable Medallion Architecture solutions (Bronze, Silver, Gold).
• Ensure data governance, lineage, security, and compliance are embedded.
• Provide technical leadership across multiple squads.
• Define processes, drive automation, and provide technical roadmaps to target state.
• Coach and mentor team members; act as escalation point for technical issues.
• Coordinate with vendors to align data roadmaps.
• Collaborate with stakeholders to meet needs and standards.
• Lead data product design and be the technical expert in data engineering tools.
• Focus on resilience, availability, and recovery for operational resilience.

Skills

Mentoring

Leadership

Stakeholder management

Project management

Tools

Databricks

PySpark

Python

Delta Lake

Databricks SQL

SQL

Databricks Genie

Job description

Principal Data Engineer

We have an exciting opportunity for an experienced Data Engineer to join our team.

This is a hybrid role, with requirements to work from either our Manchester, Stoke or Peterborough office twice a week.

Role Purpose

The Principal Data Engineer is the senior technical authority for the Databricks Lakehouse platform, responsible for defining engineering standards, architecture, and best practices across the enterprise data ecosystem.

The role leads the design and delivery of scalable, secure, high-performing data products using Databricks, Delta Lake, Unity Catalog and Azure Data Services

The role provides technical leadership, ensuring solutions are robust, cost-effective, highly available, and aligned to business strategy and operational resilience requirements.

Key Accountabilities and Responsibilities

• Being the primary technical contact on data for all new projects, designing the detailed data solution and ensuring that requirements are understood and achievable.
• Design scalable Medallion Architecture solutions (Bronze, Silver, Gold).
• Ensure data governance, lineage, security, and compliance requirements are embedded within all solutions.
• Provide technical leadership across multiple squads.
• Driving continuous improvement through defining and redefining processes within the team, increased automation and providing detailed technical roadmaps of how to achieve our target state.
• Coaching and mentoring other members of the team in technical, delivery and behavioural areas, as well as being the primary escalation point for technical issues and decisions.
• Be the technical contact with vendors and data source supplier to ensure data roadmap is aligned.
• Closely working with stakeholders to ensure they are meeting our needs and delivering to required standards.
• Taking Lead on data product design and development and be the technical expert in all data engineering toolset.
• A heavy focus on resilience, availability and recovery to provide a solid platform that meets Operational Resilience needs for all our systems.

• Proactive technical resource management including availability, capacity, performance and operability.
• Be able to deliver robust data products end to end from requirement gathering to release within time and budget.
• Create clear documentation for the data solution.
• Communicate well with stakeholder and non-technical team.
• To ensure all codes are written are in high standard and perform well.
• Provide technical training and mentor data engineers and support their personal technical development.
• To ensure data dictionaries and models are robust and maintained on a regular basis.

Technical Skill

• Expert-level Databricks experience
• Advanced PySpark development

• Delta Lake architecture and optimisation
• Databricks SQL
• Data warehouse
• Dimensional modelling
• Data Lake
• Databricks Genie
• Python development

Key Skills, Experience and Knowledge

• Extensive Data Engineering experience within enterprise-scale data environments.
• Extensive hands‑on experience delivering solutions using Databricks.
• Extensive experience designing and implementing Lakehouse architectures.
• Strong commercial experience using PySpark, Python, SQL and Delta Lake.
• Experience optimising Databricks workloads for performance, scalability and cost efficiency
• Previous experience delivering solutions within insurance, financial services or similarly regulated industries.
• Ability to understand and interpret business needs and how they translate to technical and functional requirements.
• Good project management skills; experience managing projects / requests with multiple senior stakeholders.
• Experience managing and mentoring engineering teams.

About Us

Markerstudy Group Services Limited is one of the largest Managing General Agents in the UK. With a strong presence in the UK motor insurance market, we specialise in niche motor cover, where our solid market knowledge and experience enables us to create highly targeted products.

Our success is underpinned by our underwriting strategy to identify and apply special risk factors to the customers advantage. That, and our skilled underwriting technicians who are friendly, accessible and empowered to make decisions.

Benefits

• Starting with 25 days annual leave which increases with service

• Opportunity for a yearly bonus
• Private Medical Insurance
• Life Assurance 4x annual salary
• Collaborative, fast paced working environment
• Hybrid Working
• Modern, vibrant offices