Pyspark Data Engineer

JR United Kingdom; England; Remote; GBP 60,000 - 80,000; Full time

Job summary

A leading organization is seeking a PySpark Data Engineer to develop a modern data lake as part of a strategic programme. This greenfield initiative allows for creativity in building high-impact platforms and offers a 6-month contract with potential extensions. The role involves critical data processing using PySpark and collaboration within Agile teams, ideal for engineers passionate about secure data solutions.

Qualifications

• Proven expertise in PySpark 3/4 and Python 3 for large-scale data engineering.
• Active SC clearance is essential.

Responsibilities

• Design, build, and maintain scalable data pipelines.
• Contribute to the creation of a unified data lake.
• Collaborate with cross-functional teams in Agile workflows.

Skills

PySpark 3/4

Python 3

Databricks

Delta Lake

Behaviour Driven Development

YAML

Git

Azure DevOps

Job description

Social network you want to login/join with:

PySpark Data Engineer | up to £450/day Inside | Remote with occasional London travel

We are seeking a PySpark Data Engineer to support the development of a modern, scalable data lake for a new strategic programme. This is a greenfield initiative to replace fragmented legacy reporting solutions, offering the opportunity to shape a long-term, high-impact platform from the ground up.

Key Responsibilities:
* Design, build, and maintain scalable data pipelines using PySpark 3/4 and Python 3.
* Contribute to the creation of a unified data lake following medallion architecture principles.
* Leverage Databricks and Delta Lake (Parquet format) for efficient, reliable data processing.
* Apply BDD testing practices using Python Behave and ensure code quality with Python Coverage.
* Collaborate with cross-functional teams and participate in Agile delivery workflows.
* Manage configurations and workflows using YAML, Git, and Azure DevOps.

Required Skills & Experience:
* Proven expertise in PySpark 3/4 and Python 3 for large-scale data engineering.
* Hands-on experience with Databricks, Delta Lake, and medallion architecture.
* Familiarity with Python Behave for Behaviour Driven Development.
* Strong understanding of YAML, code quality tools (e.g. Python Coverage), and CI/CD pipelines.
* Knowledge of Azure DevOps and Git best practices.
* Active SC clearance is essential - applicants without this cannot be considered.

Contract Details:
* 6-month initial contract with long-term extension potential (multi-year programme).
* Inside IR35.

This is an excellent opportunity to join a high-profile programme at its inception and help build a critical data platform from the ground up. If you are a mission-driven engineer with a passion for scalable data solutions and secure environments, we'd love to hear from you.