Data Engineer - Azure, DataBricks, PySpark

Arthur Recruitment; Greater London; Hybrid; GBP 68,000 - 83,000; Full time

Job summary

Arthur Recruitment is seeking a Data Engineer for a leading London Market insurer with a hybrid working pattern in London. You will design, build and maintain scalable data solutions across underwriting, finance and operations using Azure, Databricks, SQL and Python.

You will collaborate with engineers, analysts and business stakeholders to deliver robust data pipelines, improve data quality and optimise cloud-based data processing.

Qualifications

• Commercial experience as a Data Engineer.
• Experience with Python and/or PySpark.
• Experience with Microsoft Azure data technologies.
• Exposure to Databricks or a strong desire to develop expertise in the platform.
• Experience building ETL or ELT data pipelines.
• Knowledge of Azure Data Factory or similar integration tools.
• Understanding of Git, source control and CI/CD practices.
• Experience working within Agile delivery environments.
• Strong analytical and problem-solving skills.
• Excellent communication and stakeholder engagement skills.
• A proactive mindset with a genuine enthusiasm for learning new technologies.

Responsibilities

• Designing and developing cloud-based data pipelines using Azure Data Factory and Databricks.
• Building scalable ETL and ELT solutions to ingest and transform data from multiple business systems.
• Developing data solutions using SQL, Python and PySpark.
• Supporting the deployment of production-ready data pipelines using CI/CD and DevOps best practices.
• Working with business stakeholders to understand data requirements and deliver technical solutions.
• Improving data quality, performance and reliability across the data platform.
• Supporting data modelling, orchestration and cloud data architecture initiatives.
• Troubleshooting production issues and continuously improving engineering standards.
• Collaborating with the wider Data & Analytics team to deliver high-value data products.

Skills

Python

PySpark

SQL

Azure

Databricks

CI/CD

Agile

Git

Stakeholder engagement

Communication

Tools

Databricks

Azure Data Factory

Git

Job description

London | Hybrid

Up to £75,000 + 25% Bonus + Excellent Benefits

A leading London Market insurer is continuing to invest in its enterprise data platform and is looking for a Data Engineer to join a growing Data & Analytics function.

The Opportunity

Join a growing Data & Analytics team within a leading London Market insurer investing heavily in its cloud data platform. Working with modern technologies including Azure, Databricks, SQL and Python, you'll play a key role in designing, building and maintaining scalable data solutions that enable smarter decision making across Underwriting, Finance and Operations.

You'll collaborate with engineers, analysts and business stakeholders to develop robust data pipelines, improve data quality and optimise cloud-based data processing. This is an excellent opportunity to work on enterprise-scale data engineering projects while helping shape the future of a modern Azure and Databricks platform, with plenty of scope to develop your technical skills and progress your career.

What You'll Be Doing

• Designing and developing cloud-based data pipelines using Azure Data Factory and Databricks.
• Building scalable ETL and ELT solutions to ingest and transform data from multiple business systems.
• Developing data solutions using SQL, Python and PySpark.
• Supporting the deployment of production-ready data pipelines using CI/CD and DevOps best practices.
• Working with business stakeholders to understand data requirements and deliver technical solutions.
• Improving data quality, performance and reliability across the data platform.
• Supporting data modelling, orchestration and cloud data architecture initiatives.
• Troubleshooting production issues and continuously improving engineering standards.
• Collaborating with the wider Data & Analytics team to deliver high-value data products.

What We're Looking For

• Commercial experience as a Data Engineer.
• Experience with Python and/or PySpark.
• Experience working with Microsoft Azure data technologies.
• Exposure to Databricks or a strong desire to develop expertise in the platform.
• Experience building ETL or ELT data pipelines.
• Knowledge of Azure Data Factory or similar integration tools.
• Understanding of Git, source control and CI/CD practices.
• Experience working within Agile delivery environments.
• Strong analytical and problem-solving skills.
• Excellent communication and stakeholder engagement skills.
• A proactive mindset with a genuine enthusiasm for learning new technologies.