Tech Lead (Azure Databricks Data Platform)

Aubay UK; Greater London; On-site; GBP 90,000 - 130,000; Full time

Job summary

Aubay UK is seeking an experienced Tech Lead to drive the design, development and delivery of enterprise-scale data platform solutions on the Azure Data Platform and Databricks ecosystem.

The role requires hands-on expertise in Azure Databricks, Structured Streaming, Delta Live Tables, PySpark, Python, and SQL, along with strong leadership and stakeholder management capabilities. You will mentor teams and optimise platform performance and costs.

Qualifications

• Hands-on experience with Azure Databricks, PySpark, Python and SQL.
• Experience in financial energy trading environments.
• Understanding of trading data models and processes.
• Experience building enterprise-scale data platforms on Microsoft Azure.
• Expertise in Structured Streaming and Delta Live Tables (DLT).
• Ability to design scalable, high-performance data solutions.
• Strong data governance, quality, and engineering practices.
• Experience leading technical delivery from design to production.
• Strong stakeholder management and communication.
• Experience mentoring engineering teams.

Responsibilities

• Lead the design, development, and implementation of scalable data platform solutions using Azure and Databricks.
• Provide technical leadership across the full solution lifecycle, from architecture through delivery and production support.
• Define and promote best practices for data engineering, governance, security, quality, and platform standards.
• Design and optimise real-time data processing solutions using Structured Streaming and Delta Live Tables (DLT).
• Collaborate with business stakeholders to translate business requirements into technical solutions.
• Drive technical decision-making and ensure alignment with platform strategy and business objectives.
• Monitor and optimise platform performance, reliability, scalability, and cloud costs.
• Support the development of data architecture, integration patterns, and engineering roadmaps.
• Mentor Data Engineers and provide technical guidance across delivery teams.
• Ensure successful delivery of business outcomes through strong ownership, accountability, and leadership.
• Work closely with architects, product teams, and stakeholders to manage technical risks, dependencies, and priorities.
• Promote continuous improvement and adoption of modern data engineering practices across the platform team.

Skills

Azure Databricks

PySpark

Python

SQL

Leadership

Stakeholder management

Data governance

Architecture design

Mentoring engineers

Delivery leadership

Tools

Azure

Databricks

Delta Live Tables

Structured Streaming

PySpark

Python

SQL

Job description

Aubay UK is seeking an experienced Tech Lead (Azure Databricks Data Platform) to lead the design, development, and delivery of enterprise-scale data platform solutions leveraging the Azure Data Platform and Databricks ecosystem.

The successful candidate will be responsible for providing end-to-end technical leadership across data engineering initiatives, driving solution architecture, platform strategy, engineering standards, and delivery excellence. Working closely with business stakeholders, architects, product teams, and engineers, the Tech Lead will translate complex business requirements into scalable, high-performance data solutions that support critical business operations and analytics capabilities.

This role requires deep hands-on expertise in Azure Databricks, Structured Streaming, Delta Live Tables (DLT), PySpark, Python, and SQL, alongside strong stakeholder management and leadership skills. The ideal candidate will have a proven track record of delivering large-scale cloud data platforms, optimising platform performance and costs, and mentoring engineering teams to achieve successful business outcomes.

Required Skills and Experience:

• Strong hands-on experience with Azure Databricks, PySpark, Python, and SQL
• Experience within financial trading, commodities trading, or energy trading environments
• Understanding of trading data models and trading business processes
• Experience building and supporting enterprise-scale data platforms on Microsoft Azure
• Expertise in Structured Streaming and Delta Live Tables (DLT)
• Experience designing scalable, high-performance data solutions and architectures
• Strong understanding of data governance, data quality, and engineering best practices
• Experience leading technical delivery from design through to production support
• Strong stakeholder management and communication skills
• Experience leading and mentoring engineering teams

Desired Skills and Experience:

• Experience optimising Databricks performance and managing cloud platform costs
• Knowledge of Lakehouse architecture and modern data platform design patterns
• Experience working within Agile delivery environments
• Relevant Azure and/or Databricks certifications

Roles and Responsibilities:

• Lead the design, development, and implementation of scalable data platform solutions using Azure and Databricks
• Provide technical leadership across the full solution lifecycle, from architecture through delivery and production support
• Define and promote best practices for data engineering, governance, security, quality, and platform standards
• Design and optimise real-time data processing solutions using Structured Streaming and Delta Live Tables (DLT)
• Collaborate with business stakeholders to translate business requirements into technical solutions
• Drive technical decision-making and ensure alignment with platform strategy and business objectives
• Monitor and optimise platform performance, reliability, scalability, and cloud costs
• Support the development of data architecture, integration patterns, and engineering roadmaps
• Mentor Data Engineers and provide technical guidance across delivery teams
• Ensure successful delivery of business outcomes through strong ownership, accountability, and leadership
• Work closely with architects, product teams, and stakeholders to manage technical risks, dependencies, and priorities
• Promote continuous improvement and adoption of modern data engineering practices across the platform team

Aubay UK – Ahead of Innovation!

Aubay UK is a recognised InSourcing Partner for client-side deployment delivered across London. Our team, based in Canary Wharf, specialises in hiring IT professionals within London’s Energy and FinTech sectors, helping our clients to expand their operations with top-tier talent who are experts in their fields. We work exclusively with clients who are globally recognized as Energy Super Majors/Financial Services and innovative FinTech players.

Aubay UK is the most recently started branch of Aubay Group www.aubay.com. Aubay Group is an international Digital Services Company, listed on a Euronext Stock Exchange, who have been operating for 25 years in the European market and working alongside some of the biggest names in the Banking, Finance, Insurance, Energy, IT/Digital, Manufacturing, Transport and Telecoms sectors. With over 7,800 employees across 7 countries and 16 offices in England, France, Belgium, Luxembourg, Italy, Spain, and Portugal, Aubay Group generated revenues of €534 million in 2023.

Our Client

Our client is one of the Super Major global energy companies with around 84,000 employees across 70+ countries who are working to power progress through cleaner energy solutions.

Specialities: Upstream/Downstream, Biofuels, Integrated Gas, New Energies, Chemicals, Energy and Trading.

You will have the opportunity to work in a challenging but rewarding environment that is fast-paced and changing fundamentally and work towards transforming the business of a Super Major energy company to meet the ambition to be a net-zero emissions energy business by 2050 whilst delivering a world-class business case that has a robust societal license to operate. In your role, you will be expected to enact change and deliver value globally across business lines and geographies.