Staff Data Engineer: AI-Driven Spark on AWS

WeDo; Greater London; Hybrid; GBP 176,000 - 230,000; Part time

Job summary

WeDo is seeking a Staff Data Engineer on a contract basis to lead data engineering modernisation for a major UK financial services client. The role is hybrid with 3 days on-site in London, paying up to £1,100 per day and Inside IR35.

The initial contract is 6 months. You will combine hands-on Python/PySpark engineering with architectural thinking and AI-enabled development techniques to deliver large-scale data migrations and new data products.

Qualifications

• Hands-on Python/PySpark and Spark performance optimization experience.
• Strong understanding of Spark internals and performance tuning.
• In-depth AWS data engineering experience, especially Glue.
• Proficient in Apache Airflow and workflow orchestration.
• Experience delivering large-scale data migrations or modernisation programmes.
• Proven ability to operate at Staff or Senior Lead Engineer level with hands-on delivery.

Responsibilities

• Provide Staff-level technical leadership across complex data engineering initiatives.
• Remain hands-on with Python, PySpark and Spark, building and validating production code.
• Design and optimise scalable data pipelines and processing frameworks on AWS.
• Tackle Spark performance issues (joins, shuffles, memory, skew, partitioning).
• Interact with AWS Glue, Airflow and related technologies to shape architecture.
• Review code to identify performance bottlenecks and improve quality.
• Establish reusable engineering patterns to accelerate large-scale migrations.
• Apply AI-assisted development tools to boost productivity and delivery speed.

Skills

Python

PySpark

Apache Spark

Spark performance

AWS data engineering

Apache Airflow

AI-assisted development tools

Spark internals

Code review

Tools

AWS Glue

Job description

WeDo is seeking a Staff Data Engineer on a contract basis to lead data engineering modernisation for a major UK financial services client. The role is hybrid with 3 days on-site in London, paying up to £1,100 per day and Inside IR35.

The initial contract is 6 months. You will combine hands-on Python/PySpark engineering with architectural thinking and AI-enabled development techniques to deliver large-scale data migrations and new data products.