Lead Databricks Engineer

Simpson Associates; York and North Yorkshire; On-site; GBP 80,000 - 110,000; Full time

Job summary

Simpson Associates seeks a Lead Data Engineer in the UK to lead end-to-end data engineering deliveries using Databricks and Microsoft Fabric, guiding delivery teams and shaping technical designs.

The role requires strong consultancy, communication and leadership, with hands-on delivery and mentoring junior consultants to ensure projects meet time, cost and quality targets while aligning with client architectures.

Qualifications

• Strong experience designing and delivering modern data engineering solutions on Azure.
• Hands-on expertise with Microsoft Fabric, including Lakehouse, Data Engineering and orchestration capabilities.
• Strong experience with Databricks, including Spark-based data processing and optimisation.
• Solid understanding of data modelling, data integration patterns, and ELT/ETL pipelines.
• Experience working with Azure data services such as Azure Data Lake, Azure SQL and related platform components.
• Knowledge of DevOps and deployment practices for data platforms (e.g. CI/CD, version control, automated testing).

Responsibilities

• Lead the end-to-end delivery of data engineering projects, either independently or as part of larger multi-disciplinary programmes.
• Act as the primary service delivery contact for customers, building trusted relationships and providing clear technical leadership.
• Demonstrate deep technical expertise in data engineering, understanding how solutions fit within wider customer architectures and business requirements.
• Work with customers to identify, challenge and prioritise business and technical requirements, including running design and discovery workshops.
• Translate functional and non-functional requirements into robust, scalable data platform architectures and technical designs.
• Design and implement data engineering solutions using Microsoft Fabric and Databricks, following best-practice approaches.
• Produce clear technical documentation and communicate designs using appropriate visualisations (e.g. architecture diagrams, data flows).
• Accurately estimate delivery effort, contributing to proposals and addressing variances between pre-sales estimates and detailed requirements.
• Convert estimates into delivery plans and manage these throughout the project lifecycle, working with Project Management and Scheduling as required.
• Provide hands-on technical delivery alongside other consultants when appropriate.
• Take overall responsibility for solution and delivery quality, including code reviews, testing strategies and customer acceptance.
• Identify, manage and mitigate delivery risks and issues early, drawing on experience to pre-empt likely challenges.
• Ensure appropriate deployment, handover and transition into support or managed services.
• Mentor and support junior consultants, contributing to capability development within the Data Engineering team.
• Contribute to intellectual property, reusable assets, marketing content, case studies and blogs.

Skills

Azure data engineering

Microsoft Fabric

Databricks

Spark

ELT/ETL

Azure Data Lake

Azure SQL

DevOps CI/CD

Version control

Automated testing

Tools

Databricks

Job description

Simpson Associates transforms raw data into actionable insights that drive positive change.

Our Microsoft data expertise, our specialist sector knowledge, plus our innovative and trusted advice and guidance are just some of the reasons clients choose to work with us.

Our mission is to help purpose-led organisations from within the public and private sectors to harness data as a lever for change and enable them to realise business value more quickly. We provide the full range of services to support organisations on their data transformation journey. From advisory support and data strategy, to developing Data & AI solutions, right through to providing a range of managed services.

We are a Microsoft Solutions Partner for Cloud & AI Platforms and Security, holding Specialisations in AI Platform on Microsoft Azure, Build AI Apps on Microsoft Azure, Analytics on Microsoft Azure and Data Warehouse Migration to Microsoft Azure. Previously, Cloud & AI platforms were individual pillars for Data & AI, Digital & App innovation and Infrastructure.

But it's not just about the badges. We are proud to be recognised as the winner of the 2024 Microsoft Community Response Partner of the Year award, reflecting our dedication to using technology for positive change. We are also a Databricks partner, and an IBM Gold Partner, specialising in Cognos Analytics and Planning Analytics.

With offices in York and Sheffield, and a team based throughout the UK – we champion creativity, innovation and collaboration in the workplace.

The Role

The Lead Data Engineer plays a pivotal role within the Delivery capabilities delivering high-quality solutions using Databricks, Microsoft Fabric and Azure-native technologies. The role may involve running projects independently or leading workstreams within larger and more complex engagements under the direction of a Principal Consultant.

The Lead Data Engineer is the primary point of contact for customers during delivery and is expected to combine strong technical capability with excellent consultancy, communication and leadership skills. You will be responsible for shaping solutions, guiding delivery teams, and ensuring projects are delivered to time, cost and quality expectations, while supporting the development of junior consultants.

Key Responsibilities

• Lead the end-to-end delivery of data engineering projects, either independently or as part of larger multi-disciplinary programmes.
• Act as the primary service delivery contact for customers, building trusted relationships and providing clear technical leadership.
• Demonstrate deep technical expertise in data engineering, understanding how solutions fit within wider customer architectures and business requirements.
• Work with customers to identify, challenge and prioritise business and technical requirements, including running design and discovery workshops.
• Translate functional and non-functional requirements into robust, scalable data platform architectures and technical designs.
• Design and implement data engineering solutions using Microsoft Fabric and Databricks, following best-practice approaches.
• Produce clear technical documentation and communicate designs using appropriate visualisations (e.g. architecture diagrams, data flows).
• Accurately estimate delivery effort, contributing to proposals and addressing variances between pre-sales estimates and detailed requirements.
• Convert estimates into delivery plans and manage these throughout the project lifecycle, working with Project Management and Scheduling as required.
• Provide hands-on technical delivery alongside other consultants when appropriate.
• Take overall responsibility for solution and delivery quality, including code reviews, testing strategies and customer acceptance.
• Identify, manage and mitigate delivery risks and issues early, drawing on experience to pre-empt likely challenges.
• Ensure appropriate deployment, handover and transition into support or managed services.
• Mentor and support junior consultants, contributing to capability development within the Data Engineering team.
• Contribute to intellectual property, reusable assets, marketing content, case studies and blogs.

Skills and Attributes Required

Technical Skills

• Strong experience designing and delivering modern data engineering solutions on Azure.
• Hands-on expertise with Microsoft Fabric, including Lakehouse, Data Engineering and orchestration capabilities.
• Strong experience with Databricks, including Spark-based data processing and optimisation.
• Solid understanding of data modelling, data integration patterns, and ELT/ETL pipelines.
• Experience working with Azure data services such as Azure Data Lake, Azure SQL and related platform components.
• Knowledge of DevOps and deployment practices for data platforms (e.g. CI/CD, version control, automated testing).

Consulting & Delivery Skills

• Proven experience delivering projects within agreed time, cost and quality constraints.
• Strong ability to communicate complex technical concepts clearly to both technical and non-technical stakeholders.
• Experience leading delivery in both waterfall and agile environments and adapting approach to suit the customer and engagement.
• Commercial awareness and consultative mindset, with the ability to identify additional opportunities and add value for customers.

Personal Attributes

• Confident, proactive problem-solver who remains calm and reassuring in customer-facing situations.
• Strong organisational and time management skills, with the ability to manage multiple priorities simultaneously.
• A proven self-starter, comfortable working independently while contributing positively to team delivery.
• Commitment to continuous professional development and maintaining up-to-date technical and consultancy skills.
• Willingness to mentor others and contribute to a collaborative, knowledge-sharing culture.

Due to the nature of our work, successful applicants will need to successfully complete the relevant security clearance.

Simpson Associates reserves the right to close the recruitment process at any time.