Lead Data Platform Engineer

Irwin Mitchell; Birmingham; Hybrid; GBP 60,000 - 80,000; Full time

Job summary

A prestigious law firm in Birmingham is seeking a Lead Data Platform Engineer to architect, design, and implement scalable data platforms. This role combines engineering with strong solution architecture skills, focusing on tools like Databricks and Azure Data Factory, alongside DevOps practices. Ideal candidates will have experience with cloud environments, strong coding abilities in PySpark and Python, and a passion for mentoring. The firm offers excellent benefits including flexible holidays and health memberships.

Benefits

25 days holidays plus bank holidays

Generous pension schemes

Volunteering days

Health membership

Qualifications

• Proven experience designing and implementing data platforms in cloud environments, particularly Azure.
• Strong hands-on experience with Databricks, PySpark, Spark SQL, and Azure Data Factory.
• Advanced expertise in DevOps practices for CI/CD and automated deployments.

Responsibilities

• Architect and design end-to-end data platform solutions, ensuring scalability and reliability.
• Lead implementation using Databricks, PySpark, and Azure Data Factory.
• Monitor and optimise platform performance and manage cloud costs.

Skills

Databricks

PySpark

Spark SQL

Python

RESTful API development

CI/CD

Infrastructure-as-code

Tools

Azure Data Factory

Terraform

ARM templates

Azure DevOps

Job description

Role Overview

We are seeking a highly skilled Lead Data Platform Engineer to join our Data Engineering and Machine Learning team. This role is pivotal in designing, architecting, and delivering robust, scalable, and secure data platforms that enable the firm to manage, analyse, and leverage data effectively while meeting regulatory and client confidentiality requirements.

You will combine hands-on engineering with strong solution architecture skills, ensuring that data platform solutions are fit-for-purpose, well-governed, and aligned to business needs. A key focus will be on Databricks, Azure Data Factory, and the Lakehouse Medallion architecture, with DevOps and automation at the heart of everything you do.

About You

Key Responsibilities

• Architect and design end-to-end data platform solutions, ensuring scalability, reliability, and compliance.
• Lead implementation using Databricks, PySpark, Spark SQL, and Azure Data Factory.
• Develop APIs for data integration and automation.
• Write efficient, maintainable code in PySpark, Python, and SQL.
• Implement and manage CI/CD pipelines and automated deployments via Azure DevOps.
• Build infrastructure-as-code solutions (Terraform, ARM templates) for cloud resource provisioning.
• Monitor and optimise platform performance and manage cloud costs.
• Ensure data quality, security, governance, and lineage across all components.
• Collaborate with data engineers, architects, and business stakeholders to translate requirements into effective solutions.
• Maintain comprehensive documentation and stay current with emerging technologies.
• Provide coaching and mentoring to engineers, fostering a culture of continuous learning and technical excellence.

Essential Skills

• Proven experience designing and engineering data platforms in cloud environments (preferably Azure).
• Strong hands‑on experience with Databricks, PySpark, Spark SQL, and Azure Data Factory.
• Proficiency in Python and RESTful API development.
• Advanced expertise in DevOps practices for CI/CD and automated deployments.
• Experience with infrastructure-as-code tools (Terraform, ARM templates).

Desirable Skills

• Experience with additional Azure services (Fabric, Functions, Logic Apps).
• Data Lakehouse architecture.
• Background in regulated or professional services environments.

Our Benefits - What We Can Offer You

• 25 days holidays as standard plus bank holidays - You can ‘buy’ up to 35hrs of extra holiday too.
• Generous and flexible pension schemes.
• Volunteering days – Two days of volunteering every year for a cause of your choice (fully paid)
• Westfield Health membership, offering refunds on medical services alongside our Aviva Digital GP services.

We also offer a wide range of well‑being initiatives to encourage positive mental health both in and out of the workplace and to make sure you’re fully supported. This includes our Flexible by Choice programme which gives our colleagues more choice over a hybrid way of working subject to role, team and client requirements.

We have been ranked in the Best Workplaces for Wellbeing for Large Organisations for 2024!

Our responsible business programmes are fundamental to who we are and our purpose. We’re committed to being a diverse and inclusive workplace where our colleagues can flourish, and we have established a number of inclusion network groups across our business to support this aim.

Our commitment to Social Responsibility, community investment activity and tackling climate change is a fundamental part of who we are. It’s made up of four strands: Our People, Our Community, Our Environment and Our Pro Bono.