Data Engineer - Azure, DataBricks, PySpark

Arthur Recruitment; Greater London; Hybrid; GBP 59,000 - 72,000; Full time

Job summary

London Market insurer is seeking a Data Engineer to join the Data & Analytics team in London. You will design, build and maintain scalable data solutions on Azure and Databricks, enabling smarter decision making across underwriting, finance and operations.

You'll work with engineers and business stakeholders to develop robust data pipelines, improve data quality and apply CI/CD practices in a cloud-first environment. Hybrid work with strong growth opportunities.

Qualifications

• Commercial experience as a Data Engineer.
• Experience with Python and/or PySpark.
• Experience with Microsoft Azure data technologies.
• Exposure to Databricks or a strong desire to develop expertise in the platform.
• Experience building ETL or ELT data pipelines.
• Knowledge of Azure Data Factory or similar integration tools.
• Understanding of Git, source control and CI/CD practices.
• Experience working within Agile delivery environments.
• Strong analytical and problem-solving skills.
• Excellent communication and stakeholder engagement skills.
• A proactive mindset with a genuine enthusiasm for learning new technologies.

Responsibilities

• Designing and developing cloud-based data pipelines using Azure Data Factory and Databricks.
• Building scalable ETL and ELT solutions to ingest and transform data from multiple business systems.
• Developing data solutions using SQL, Python and PySpark.
• Supporting the deployment of production-ready data pipelines using CI/CD and DevOps best practices.
• Working with business stakeholders to understand data requirements and deliver technical solutions.
• Improving data quality, performance and reliability across the data platform.
• Supporting data modelling, orchestration and cloud data architecture initiatives.
• Troubleshooting production issues and continuously improving engineering standards.
• Collaborating with the wider Data & Analytics team to deliver high-value data products.

Skills

Communication

Stakeholder engagement

Analytical thinking

Proactive learner

Tools

Azure

Databricks

SQL

Python

Azure Data Factory

Git

CI/CD

Job description

London | Hybrid

Up to £65,000 + 10% Bonus + Excellent Benefits

A leading London Market insurer is continuing to invest in its enterprise data platform and is looking for a Data Engineer to join a growing Data & Analytics function.

The Opportunity

Join a growing Data & Analytics team within a leading London Market insurer investing heavily in its cloud data platform. Working with modern technologies including Azure, Databricks, SQL and Python, you'll play a key role in designing, building and maintaining scalable data solutions that enable smarter decision making across Underwriting, Finance and Operations.

You'll collaborate with engineers, analysts and business stakeholders to develop robust data pipelines, improve data quality and optimise cloud-based data processing. This is an excellent opportunity to work on enterprise-scale data engineering projects while helping shape the future of a modern Azure and Databricks platform, with plenty of scope to develop your technical skills and progress your career.

What You'll Be Doing

• Designing and developing cloud-based data pipelines using Azure Data Factory and Databricks.
• Building scalable ETL and ELT solutions to ingest and transform data from multiple business systems.
• Developing data solutions using SQL, Python and PySpark.
• Supporting the deployment of production-ready data pipelines using CI/CD and DevOps best practices.
• Working with business stakeholders to understand data requirements and deliver technical solutions.
• Improving data quality, performance and reliability across the data platform.
• Supporting data modelling, orchestration and cloud data architecture initiatives.
• Troubleshooting production issues and continuously improving engineering standards.
• Collaborating with the wider Data & Analytics team to deliver high-value data products.

What We're Looking For

• Commercial experience as a Data Engineer.
• Experience with Python and/or PySpark.
• Experience working with Microsoft Azure data technologies.
• Exposure to Databricks or a strong desire to develop expertise in the platform.
• Experience building ETL or ELT data pipelines.
• Knowledge of Azure Data Factory or similar integration tools.
• Understanding of Git, source control and CI/CD practices.
• Experience working within Agile delivery environments.
• Strong analytical and problem-solving skills.
• Excellent communication and stakeholder engagement skills.
• A proactive mindset with a genuine enthusiasm for learning new technologies.