Pyspark Data Engineer

iO Associates; England; Remote; GBP 60,000 - 80,000; Full time

Job summary

A leading company seeks a PySpark Data Engineer to support the development of a modern, scalable data lake for a new strategic programme. This role offers a chance to shape a long-term platform from the ground up, engaging in Agile delivery while leveraging state-of-the-art technologies.

Qualifications

• Proven experience in PySpark 3/4 and Python 3 for data engineering.
• Hands-on with Databricks and medallion architecture.
• Active SC clearance is essential.

Responsibilities

• Design, build, and maintain scalable data pipelines.
• Leverage Databricks for reliable data processing.
• Collaborate with cross-functional teams.

Skills

PySpark 3/4

Python 3

Databricks

Delta Lake

YAML

CI/CD Pipelines

Agile Delivery

Tools

Azure DevOps

Git

Job description

PySpark Data Engineer | up to £450/day Inside | Remote with occasional London travel

We are seeking a PySpark Data Engineer to support the development of a modern, scalable data lake for a new strategic programme. This is a greenfield initiative to replace fragmented legacy reporting solutions, offering the opportunity to shape a long-term, high-impact platform from the ground up.

Key Responsibilities:
* Design, build, and maintain scalable data pipelines using PySpark 3/4 and Python 3.
* Contribute to the creation of a unified data lake following medallion architecture principles.
* Leverage Databricks and Delta Lake (Parquet format) for efficient, reliable data processing.
* Apply BDD testing practices using Python Behave and ensure code quality with Python Coverage.
* Collaborate with cross-functional teams and participate in Agile delivery workflows.
* Manage configurations and workflows using YAML, Git, and Azure DevOps.

Required Skills & Experience:
* Proven expertise in PySpark 3/4 and Python 3 for large-scale data engineering.
* Hands-on experience with Databricks, Delta Lake, and medallion architecture.
* Familiarity with Python Behave for Behaviour Driven Development.
* Strong understanding of YAML, code quality tools (e.g. Python Coverage), and CI/CD pipelines.
* Knowledge of Azure DevOps and Git best practices.
* Active SC clearance is essential - applicants without this cannot be considered.

Contract Details:
* 6-month initial contract with long-term extension potential (multi-year programme).
* Inside IR35.

This is an excellent opportunity to join a high-profile programme at its inception and help build a critical data platform from the ground up. If you are a mission-driven engineer with a passion for scalable data solutions and secure environments, we'd love to hear from you.