Staff Data Engineer | Contract | Up to £1,100/day | Inside IR35

WeDo; Greater London; Hybrid; GBP 176,000 - 230,000; Part time

Job summary

WeDo is seeking a Staff Data Engineer on a contract basis to lead data engineering modernisation for a major UK financial services client. The role is hybrid with 3 days on-site in London, paying up to £1,100 per day and Inside IR35.

The initial contract is 6 months. You will combine hands-on Python/PySpark engineering with architectural thinking and AI-enabled development techniques to deliver large-scale data migrations and new data products.

Qualifications

• Hands-on Python/PySpark and Spark performance optimization experience.
• Strong understanding of Spark internals and performance tuning.
• In-depth AWS data engineering experience, especially Glue.
• Proficient in Apache Airflow and workflow orchestration.
• Experience delivering large-scale data migrations or modernisation programmes.
• Proven ability to operate at Staff or Senior Lead Engineer level with hands-on delivery.

Responsibilities

• Provide Staff-level technical leadership across complex data engineering initiatives.
• Remain hands-on with Python, PySpark and Spark, building and validating production code.
• Design and optimise scalable data pipelines and processing frameworks on AWS.
• Tackle Spark performance issues (joins, shuffles, memory, skew, partitioning).
• Interact with AWS Glue, Airflow and related technologies to shape architecture.
• Review code to identify performance bottlenecks and improve quality.
• Establish reusable engineering patterns to accelerate large-scale migrations.
• Apply AI-assisted development tools to boost productivity and delivery speed.

Skills

Python

PySpark

Apache Spark

Spark performance

AWS data engineering

Apache Airflow

AI-assisted development tools

Spark internals

Code review

Tools

AWS Glue

Job description

Staff Data Engineer | Contract | Up to £1,100/day | Inside IR35

Location: Hybrid – London (3 days per week on-site)

Rate: Up to £1,100/day

Contract: Initial 6 months

IR35: Inside IR35

We're supporting a major UK financial services organisation through a significant programme of data engineering modernisation and transformation.

We're looking for a genuinely Staff-level Data Engineer who combines exceptional hands‑on Python/PySpark engineering with strong architectural thinking and practical experience of AI‑assisted software development.

This is not a hands‑off architecture role. You need to be equally comfortable setting technical direction, challenging engineering approaches and independently writing, debugging and optimising production‑quality code.

The sweet spot is someone who can operate at a strategic level while remaining an outstanding hands‑on engineer.

The roleYou'll play a key role in modernising a large-scale enterprise data estate, including the migration and transformation of thousands of data pipelines, development of new data products and adoption of modern, AI‑assisted engineering practices.

Responsibilities will include:

• Providing Staff-level technical leadership across complex data engineering and modernisation initiatives.
• Remaining deeply hands‑on with Python, PySpark and Apache Spark, independently building and proving engineering solutions.
• Designing and optimising scalable data pipelines and processing frameworks across AWS.
• Tackling complex Spark performance challenges, including shuffles, joins, data skew, partitioning and memory optimisation.
• Designing and challenging engineering approaches across AWS Glue, Apache Airflow and associated technologies.
• Reviewing existing code, identifying performance bottlenecks and driving improvements in engineering quality.
• Establishing reusable engineering patterns, frameworks and approaches to accelerate large‑scale data migrations.
• Applying AI‑assisted development tools to improve engineering productivity, code quality and delivery speed.
• Challenging technical assumptions, influencing architectural decisions and raising engineering standards across multiple teams.

What we're looking for

Core engineering capability

• Exceptional hands‑on Python/PySpark and Apache Spark expertise, with the ability to independently write, debug and optimise complex production code.
• Deep understanding of Spark internals and performance optimisation, including execution plans, joins, shuffles, partitioning, memory management and data skew.
• Strong AWS data engineering, particularly AWS Glue.
• Strong Apache Airflow experience.
• Excellent software engineering fundamentals, including code quality, testing, performance and maintainability.
• Experience designing and delivering large‑scale data pipeline migrations or modernisation programmes.

Technical leadership and AI-enabled engineering

• Experience operating at Staff, Principal or Senior Lead Engineer level, or demonstrably performing at that standard.
• Ability to independently assess complex engineering challenges, challenge existing approaches and shape scalable solutions.
• Strong architectural and conceptual understanding, underpinned by genuine hands‑on delivery experience.
• Demonstrable experience using AI‑assisted development tools (such as Claude, GitHub Copilot or similar) to accelerate real‑world software engineering delivery.
• Ability to critically review, validate and improve AI‑generated code rather than relying on it without understanding.
• Experience influencing engineering practices and technical direction across multiple teams.

Beneficial experience

• Snowflake.
• Legacy ETL modernisation and migration.
• Building reusable migration frameworks or automation to accelerate large‑scale transformation programmes.
• Financial services or other complex, highly regulated environments.

What makes someone successful in this role? We're looking for a relatively rare combination of capabilities: A genuinely excellent hands‑on Spark engineer who can think architecturally, solve complex problems independently and use AI to accelerate engineering delivery. You'll need to be comfortable discussing high‑level technical strategy one minute and getting directly into complex PySpark code the next. This is an opportunity to have a meaningful impact on a major enterprise transformation programme while remaining at the forefront of hands‑on engineering.