Lead Data Engineer

ARC IT Recruitment; Greater London; On-site; GBP 90,000 - 130,000; Full time

Job summary

ARC IT Recruitment is seeking a Lead Data Engineer in London, working hybrid with a prestigious wealth management organisation. This role focuses on designing, developing, and maintaining modern data solutions using Microsoft Fabric and Azure technologies to support business-critical decisions.

You will define engineering standards, optimise performance, and mentor engineers while collaborating with data scientists, analysts, and stakeholders to deliver high-value data products and ensure

Qualifications

• 5+ years' experience in Data Engineering within cloud environments.
• Expertise with Microsoft Fabric and Azure technologies.
• Advanced Python and SQL development skills.
• Experience building and optimising ETL/ELT pipelines.
• Knowledge of data modelling, semantic models, and enterprise reporting solutions.
• Experience implementing data governance, security, and quality frameworks.

Responsibilities

• Define and implement data engineering standards, frameworks, and best practices.
• Design, build, and maintain secure, scalable, and observable data pipelines.
• Develop data solutions using Microsoft Fabric, Lakehouse, and Data Warehouse architectures.
• Implement and manage Medallion Architecture patterns for data ingestion and transformation.
• Ensure data quality, completeness, integrity, and governance across the platform.
• Optimise data processing workflows, pipelines, and storage solutions for performance and cost efficiency.
• Collaborate with Data Scientists, Analysts, Product Owners, and business stakeholders to deliver high-value data products.
• Support CI/CD implementation and engineering automation practices.
• Contribute to platform roadmap planning and adoption of emerging Microsoft Fabric capabilities.
• Design monitoring, disaster recovery, and business continuity strategies for critical data services.
• Promote effective data governance, metadata management, and catalogue standards.
• Mentor engineers and provide technical leadership across multiple data initiatives.

Skills

Python

SQL

Data Engineering

Microsoft Fabric

Azure

ETL/ELT

CI/CD

Data Governance

Observability

Data Platform

Tools

Microsoft Fabric

Azure Synapse

Azure Data Factory

Apache Spark

Data Lake

Data Warehouse

Lakehouse

Medallion Architecture

Job description

Lead Data Engineer

Location: London, City (Hybrid)

£ Highly competitive base with bonus and benefits

Lead Data Engineer is sought by expanding central data platform team within a prestigious global wealth management organisation. This role offers the opportunity to help shape enterprise-wide data engineering standards, contribute to platform strategy, and build modern, scalable data solutions that support business-critical decision-making. Essential to this role are strong python skills, expertise in MS Fabric, SQL, data warehousing, Data Lake and Azure technologies.

The Role

As the Lead Data Engineer, you will be responsible for designing, developing, and maintaining modern data solutions using Microsoft Fabric and Azure technologies. You will contribute to the development of engineering standards, optimise platform performance, and support the adoption of secure, scalable, and observable data architectures.

Working closely with stakeholders, product owners, analysts, data scientists, and platform specialists, you will play a key role in delivering high-quality data products and promoting best practices across federated data teams.

Key Responsibilities

• Define and implement data engineering standards, frameworks, and best practices.
• Design, build, and maintain secure, scalable, and observable data pipelines.
• Develop data solutions using Microsoft Fabric, Lakehouse, and Data Warehouse architectures.
• Implement and manage Medallion Architecture patterns for data ingestion and transformation.
• Ensure data quality, completeness, integrity, and governance across the platform.
• Optimise data processing workflows, pipelines, and storage solutions for performance and cost efficiency.
• Collaborate with Data Scientists, Analysts, Product Owners, and business stakeholders to deliver high-value data products.
• Support CI/CD implementation and engineering automation practices.
• Contribute to platform roadmap planning and adoption of emerging Microsoft Fabric capabilities.
• Design monitoring, disaster recovery, and business continuity strategies for critical data services.
• Promote effective data governance, metadata management, and catalogue standards.
• Mentor engineers and provide technical leadership across multiple data initiatives.

Required Skills & Experience

• 5 years' experience in Data Engineering within cloud-based environments.
• Proven expertise with Microsoft Fabric and modern data platform technologies.
• Advanced Python and SQL development skills.
• Experience building and optimising ETL/ELT pipelines.
• Strong understanding of Lakehouse, Data Warehouse, and Data Lake architectures.
• Experience with Azure Data Factory, Azure Synapse, and Apache Spark.
• Knowledge of data modelling, semantic models, and enterprise reporting solutions.
• Experience implementing data governance, security, and quality frameworks.
• Strong understanding of modern data engineering practices, including observability and monitoring.
• Experience working with CI/CD pipelines and DevOps practices.
• Ability to engage effectively with both technical and non-technical stakeholders.

What We're Looking For

The successful candidate will be a proactive and commercially minded data engineering leader who combines strong technical capability with excellent communication skills. You will have experience delivering complex data platform initiatives, influencing stakeholders at all levels, and helping organisations realise the value of modern data technologies.

If you are passionate about building scalable data platforms, driving engineering excellence, and shaping the future of enterprise data capabilities, we would love to hear from you.