Lead Data Engineer: Databricks & Azure Solutions

Simpson Associates; York and North Yorkshire; On-site; GBP 80,000 - 110,000; Full time

Job summary

Simpson Associates seeks a Lead Data Engineer in the UK to lead end-to-end data engineering deliveries using Databricks and Microsoft Fabric, guiding delivery teams and shaping technical designs.

The role requires strong consultancy, communication and leadership, with hands-on delivery and mentoring junior consultants to ensure projects meet time, cost and quality targets while aligning with client architectures.

Qualifications

• Strong experience designing and delivering modern data engineering solutions on Azure.
• Hands-on expertise with Microsoft Fabric, including Lakehouse, Data Engineering and orchestration capabilities.
• Strong experience with Databricks, including Spark-based data processing and optimisation.
• Solid understanding of data modelling, data integration patterns, and ELT/ETL pipelines.
• Experience working with Azure data services such as Azure Data Lake, Azure SQL and related platform components.
• Knowledge of DevOps and deployment practices for data platforms (e.g. CI/CD, version control, automated testing).

Responsibilities

• Lead the end-to-end delivery of data engineering projects, either independently or as part of larger multi-disciplinary programmes.
• Act as the primary service delivery contact for customers, building trusted relationships and providing clear technical leadership.
• Demonstrate deep technical expertise in data engineering, understanding how solutions fit within wider customer architectures and business requirements.
• Work with customers to identify, challenge and prioritise business and technical requirements, including running design and discovery workshops.
• Translate functional and non-functional requirements into robust, scalable data platform architectures and technical designs.
• Design and implement data engineering solutions using Microsoft Fabric and Databricks, following best-practice approaches.
• Produce clear technical documentation and communicate designs using appropriate visualisations (e.g. architecture diagrams, data flows).
• Accurately estimate delivery effort, contributing to proposals and addressing variances between pre-sales estimates and detailed requirements.
• Convert estimates into delivery plans and manage these throughout the project lifecycle, working with Project Management and Scheduling as required.
• Provide hands-on technical delivery alongside other consultants when appropriate.
• Take overall responsibility for solution and delivery quality, including code reviews, testing strategies and customer acceptance.
• Identify, manage and mitigate delivery risks and issues early, drawing on experience to pre-empt likely challenges.
• Ensure appropriate deployment, handover and transition into support or managed services.
• Mentor and support junior consultants, contributing to capability development within the Data Engineering team.
• Contribute to intellectual property, reusable assets, marketing content, case studies and blogs.

Skills

Azure data engineering

Microsoft Fabric

Databricks

Spark

ELT/ETL

Azure Data Lake

Azure SQL

DevOps CI/CD

Version control

Automated testing

Tools

Databricks

Job description

Simpson Associates seeks a Lead Data Engineer in the UK to lead end-to-end data engineering deliveries using Databricks and Microsoft Fabric, guiding delivery teams and shaping technical designs.

The role requires strong consultancy, communication and leadership, with hands-on delivery and mentoring junior consultants to ensure projects meet time, cost and quality targets while aligning with client architectures.