Lead Data Engineer Hadoop Spark Python

Square One Resources; Sheffield; Hybrid; GBP 80,000 - 100,000; Full time

Job summary

A leading recruitment agency is looking for a Lead Data Engineer to join their team in Sheffield. The role involves developing data engineering platforms and integrating large datasets. Ideal candidates should have extensive expertise in Hadoop and Spark, proficiency in Python, and experience working with BI teams. This contract position requires in-office work three days per week, with competitive pay of up to £617 per day until November 2026.

Qualifications

• Proven experience in data engineering platforms.
• Extensive enterprise experience with Hadoop and Spark.
• Proficiency in Python and scripting.

Responsibilities

• Lead development of data engineering platforms.
• Build and optimize ETL/ELT data pipelines.
• Collaborate with BI and Analytics teams.

Skills

Hadoop

Spark

Python

SQL

NoSQL

ETL/ELT

CI/CD

Agile methodologies

SRE culture

Job description

Job Title: Lead Data Engineer - Hadoop, Spark, Pytthon

Location: Sheffield - 3 days per week in the office

Salary/Rate: Up to £617 per day inside IR35

Start Date: 02/03/2026

Job Type: Contract until November

We are seeking a Lead Data Engineering Consultant with proven experience in leading and developing data engineering platforms.

Ideal Candidate

• Extensive enterprise experience with Hadoop, Spark, and Splunk.
• Proficiency in object-oriented and functional scripting, particularly in Python.
• Skilled in handling raw, structured, semi-structured, and unstructured data (SQL and NoSQL).
• Experience integrating large, disparate datasets using modern tools and frameworks.
• Strong background in building and optimizing ETL/ELT data pipelines.
• Familiarity with source control and implementing Continuous Integration, Delivery, and Deployment via CI/CD pipelines.
• Experience supporting and collaborating with BI and Analytics teams in fast-paced environments.
• Ability to pair program and work effectively with other engineers.
• Excellent analytical and problem-solving abilities.
• Knowledge of agile methodologies such as Scrum or Kanban is a plus.
• Comfortable representing the team in standups and problem-solving sessions.
• Capable of driving the creation of technical test plans and maintaining records, including unit and integration tests, within automated test environments to ensure high code quality.
• Promote SRE (Site Reliability Engineering) culture by addressing challenges through data engineering.
• Ensure service resilience, sustainability, and adherence to recovery time objectives for all delivered software solutions.

If you are interested in this opportunity, please apply now with your updated CV in Microsoft Word/PDF format.

Disclaimer

Notwithstanding any guidelines given to level of experience sought, we will consider candidates from outside this range if they can demonstrate the necessary competencies. Square One is acting as both an employment agency and an employment business, and is an equal opportunities recruitment business. Square One embraces diversity and will treat everyone equally. Please see our website for our full diversity statement