Data Engineer: Azure, Databricks & PySpark - Hybrid London

Arthur Recruitment; Greater London; Hybrid; GBP 59,000 - 72,000; Full time

Job summary

London Market insurer is seeking a Data Engineer to join the Data & Analytics team in London. You will design, build and maintain scalable data solutions on Azure and Databricks, enabling smarter decision making across underwriting, finance and operations.

You'll work with engineers and business stakeholders to develop robust data pipelines, improve data quality and apply CI/CD practices in a cloud-first environment. Hybrid work with strong growth opportunities.

Qualifications

• Commercial experience as a Data Engineer.
• Experience with Python and/or PySpark.
• Experience with Microsoft Azure data technologies.
• Exposure to Databricks or a strong desire to develop expertise in the platform.
• Experience building ETL or ELT data pipelines.
• Knowledge of Azure Data Factory or similar integration tools.
• Understanding of Git, source control and CI/CD practices.
• Experience working within Agile delivery environments.
• Strong analytical and problem-solving skills.
• Excellent communication and stakeholder engagement skills.
• A proactive mindset with a genuine enthusiasm for learning new technologies.

Responsibilities

• Designing and developing cloud-based data pipelines using Azure Data Factory and Databricks.
• Building scalable ETL and ELT solutions to ingest and transform data from multiple business systems.
• Developing data solutions using SQL, Python and PySpark.
• Supporting the deployment of production-ready data pipelines using CI/CD and DevOps best practices.
• Working with business stakeholders to understand data requirements and deliver technical solutions.
• Improving data quality, performance and reliability across the data platform.
• Supporting data modelling, orchestration and cloud data architecture initiatives.
• Troubleshooting production issues and continuously improving engineering standards.
• Collaborating with the wider Data & Analytics team to deliver high-value data products.

Skills

Communication

Stakeholder engagement

Analytical thinking

Proactive learner

Tools

Azure

Databricks

SQL

Python

Azure Data Factory

Git

CI/CD

Job description

London Market insurer is seeking a Data Engineer to join the Data & Analytics team in London. You will design, build and maintain scalable data solutions on Azure and Databricks, enabling smarter decision making across underwriting, finance and operations.

You'll work with engineers and business stakeholders to develop robust data pipelines, improve data quality and apply CI/CD practices in a cloud-first environment. Hybrid work with strong growth opportunities.