Lead Azure Data Engineer

Jefferson Frank; City Of London; Hybrid; GBP 60,000 - 80,000; Full time

Job summary

A leading data consultancy is seeking a Lead Azure Data Engineer to shape the future of financial reporting through innovative cloud engineering. This hybrid role based in Central London requires expertise in Azure Databricks and effective collaboration with BI and analytics teams. Ideal candidates will have a strong background in data product design, governance, and financial systems.

Qualifications

• Expertise in Azure Databricks and Spark technologies.
• Experience in scalable ETL/ELT pipeline development.
• Strong understanding of data governance and cloud architecture.

Responsibilities

• Lead design and delivery of scalable data products.
• Build and enhance ETL/ELT pipelines in Azure Databricks.
• Collaborate with cross-functional teams on data solutions.

Skills

Azure Databricks: Expert in Spark (SQL, PySpark)

Data Pipeline Design: Proven experience in scalable ETL/ELT development

Azure Services: Data Lake, Blob Storage, Synapse

Data Governance: Unity Catalog, access control, metadata management

Performance Tuning: Partitioning, caching, Spark job optimization

Cloud Architecture: Infrastructure-as-code, monitoring, automation

Finance Domain Knowledge: Experience with financial systems and reporting

Data Modelling: Kimball methodology, star schemas

Retail Experience: Preferred but not essential

Job description

Contract Opportunity: Lead Azure Data Engineer (Remote - £500/day Outside IR35)

We're hiring for a Lead Azure Data Engineer to join our team on a hybrid contract based in London, supporting key finance stakeholders and transforming our data platform. This role offers the chance to shape the future of financial reporting through cutting-edge cloud engineering and data architecture.

Location: Central London (UK-based candidates only)
Rate: £500/day
IR35 Status: Outside IR35
Start Date: ASAP - Interviews next week
Duration: 6 months with potential for extension

The Role

You'll lead the design and delivery of scalable data products that support financial analysis and decision-making. Working closely with BI and Analytics teams, you'll help evolve our data warehouse and implement best-in-class engineering practices across Azure.

Key Responsibilities

• Build and enhance ETL/ELT pipelines in Azure Databricks
• Develop facts and dimensions for financial reporting
• Collaborate with cross-functional teams to deliver robust data solutions
• Optimize data workflows for performance and cost-efficiency
• Implement governance and security using Unity Catalog
• Drive automation and CI/CD practices across the data platform
• Explore new technologies to improve data ingestion and self-service

Essential Skills

• Azure Databricks: Expert in Spark (SQL, PySpark), Databricks Workflows
• Data Pipeline Design: Proven experience in scalable ETL/ELT development
• Azure Services: Data Lake, Blob Storage, Synapse
• Data Governance: Unity Catalog, access control, metadata management
• Performance Tuning: Partitioning, caching, Spark job optimization
• Cloud Architecture: Infrastructure-as-code, monitoring, automation
• Finance Domain Knowledge: Experience with financial systems and reporting
• Data Modelling: Kimball methodology, star schemas
• Retail Experience: Preferred but not essential

About the Team

We're a collaborative data function made up of BI Developers and Data Engineers. We work end-to-end on solutions, share knowledge, and support each other's growth. Our culture values curiosity, innovation, and continuous learning.

Interested? We have limited interview slots next week and aim to fill this role by the end of the month. Please send me a copy of your CV if you meet the requirements