Pyspark Data Engineer

iO Associates - UK/EU; England; Remote; GBP 90,000 - 135,000; Full time

Job summary

A leading company is seeking a PySpark Data Engineer for a greenfield project aimed at developing a modern data lake. This contract offers the chance to work on high-impact solutions using cutting-edge technologies like Databricks and Python, contributing to a long-term strategic initiative.

Qualifications

• Active SC clearance is essential.
• Proven expertise in PySpark 3/4 and Python 3 for large-scale data engineering.
• Hands-on experience with Databricks, Delta Lake, and medallion architecture.

Responsibilities

• Design, build, and maintain scalable data pipelines using PySpark.
• Contribute to the creation of a unified data lake following medallion principles.
• Collaborate with cross-functional teams and participate in Agile workflows.

Skills

Data engineering

PySpark

Python

Databricks

YAML

Git

Azure DevOps

Behaviour Driven Development

Job description

Direct message the job poster from iO Associates - UK/EU

Placing Technical Professionals With The Best Companies Across The South - iO Associates - UK/EU

PySpark Data Engineer | up to £450/day Inside | Remote with occasional London travel

We are seeking a PySpark Data Engineer to support the development of a modern, scalable data lake for a new strategic programme. This is a greenfield initiative to replace fragmented legacy reporting solutions, offering the opportunity to shape a long-term, high-impact platform from the ground up.

Key Responsibilities:
* Design, build, and maintain scalable data pipelines using PySpark 3/4 and Python 3.
* Contribute to the creation of a unified data lake following medallion architecture principles.
* Leverage Databricks and Delta Lake (Parquet format) for efficient, reliable data processing.
* Apply BDD testing practices using Python Behave and ensure code quality with Python Coverage.
* Collaborate with cross-functional teams and participate in Agile delivery workflows.
* Manage configurations and workflows using YAML, Git, and Azure DevOps.

Required Skills & Experience:
* Proven expertise in PySpark 3/4 and Python 3 for large-scale data engineering.
* Hands-on experience with Databricks, Delta Lake, and medallion architecture.
* Familiarity with Python Behave for Behaviour Driven Development.
* Strong understanding of YAML, code quality tools (e.g. Python Coverage), and CI/CD pipelines.
* Knowledge of Azure DevOps and Git best practices.
* Active SC clearance is essential - applicants without this cannot be considered.

Contract Details:
* 6-month initial contract with long-term extension potential (multi-year programme).
* Inside IR35.

This is an excellent opportunity to join a high-profile programme at its inception and help build a critical data platform from the ground up. If you are a mission-driven engineer with a passion for scalable data solutions and secure environments, we'd love to hear from you.

Seniority level

• Seniority level

  Entry level

Employment type

• Employment type

  Contract

Job function

• Job function

  Information Technology

• Industries

  Technology, Information and Internet

Referrals increase your chances of interviewing at iO Associates - UK/EU by 2x

Get notified about new Data Engineer jobs in England, United Kingdom.

London, England, United Kingdom 3 days ago

Greater London, England, United Kingdom 1 week ago

London, England, United Kingdom 4 days ago

London, England, United Kingdom 6 days ago

London, England, United Kingdom 2 days ago

London, England, United Kingdom 3 days ago

London, England, United Kingdom 6 days ago

London, England, United Kingdom 2 weeks ago

London, England, United Kingdom 4 days ago

Leeds, England, United Kingdom 3 weeks ago

London, England, United Kingdom 5 days ago

London, England, United Kingdom 1 day ago

London, England, United Kingdom £60,000.00-£80,000.00 1 month ago

London, England, United Kingdom 14 hours ago

Greater London, England, United Kingdom 3 days ago

London, England, United Kingdom 1 week ago

Leeds, England, United Kingdom 2 weeks ago

London, England, United Kingdom 19 hours ago

London, England, United Kingdom 5 days ago

London, England, United Kingdom 1 week ago

City Of London, England, United Kingdom £75,000.00-£85,000.00 18 hours ago

City Of London, England, United Kingdom £60,000.00-£70,000.00 3 weeks ago

Greater London, England, United Kingdom 1 month ago

London, England, United Kingdom 2 weeks ago

London, England, United Kingdom 2 weeks ago

London, England, United Kingdom 5 days ago

We’re unlocking community knowledge in a new way. Experts add insights directly into each article, started with the help of AI.