URGENT AWS Data Engineer (migration)

Opus Recruitment Solutions Ltd; Greater London; Hybrid; GBP 92,000 - 97,000; Full time

Job summary

Opus Recruitment Solutions Ltd in the United Kingdom is seeking a hands-on AWS Data Engineer for a migration project. This urgent contract will move a large Ab Initio ETL estate to a modern, serverless AWS data platform.

You will define migration patterns and deliver production-ready AWS Glue solutions, working with PySpark, Python, SQL and Spark tuning on large-scale pipelines. The role offers remote-first arrangements with occasional on-site London presence.

Qualifications

• Strong AWS Data Engineer, ideally AWS Certified.
• Proven experience delivering Ab Initio to AWS Glue (PySpark) migrations.
• Hands-on AWS Data Engineering with AWS Glue, S3, Athena and serverless platforms.
• Advanced Python, PySpark and SQL skills.
• Experience converting Ab Initio transformations into scalable PySpark solutions.
• Experience building and optimizing large-scale ETL/data pipelines in AWS.
• Spark performance tuning and data skew troubleshooting.

Responsibilities

• Define migration patterns and migration strategy for Ab Initio estate.
• Develop production-ready AWS Glue solutions for the target data platform.
• Build and optimize large-scale ETL pipelines in AWS.
• Tune Spark jobs to improve performance and reduce memory/load issues.
• Collaborate with cloud and data teams in a financial services context.

Skills

AWS Data Engineer

PySpark

Python

SQL

Spark

ETL

Data Warehousing

Data Modelling

Ab Initio experience

Tools

AWS Glue

S3

Athena

Airflow (MWAA)

Job description

AWS Data Engineer - Migration

We're seeking a strong & hands-on AWS Data Engineering contractor, ASAP start, for an urgent replacement contract

Rate: Up to £500-525 per day, Outside IR35

Working Patterns: Remote-first, London (up to 1 day p/w on-site).

Contract: 3-6 months (likley extension)

Sector: Financial Services

About

We're supporting a major cloud transformation programme focused on migrating a large-scale legacy ETL estate consiting of several thousand batch jobs from Ab Initio to a modern, serverless AWS data platform. This is a hands-on Senior Data Engineer role where you'll play a key part in defining migration patterns and delivering production-ready AWS Glue solutions.

Person Requirments

• Strong AWS Data Engineer - ideally AWS Certified.
• Idelaly, you'll bring proven experience delivering Ab Initio AWS Glue (PySpark) migration programmes
• Strong hands-on AWS Data Engineering expertise, particularly AWS Glue, S3, Athena and serverless data platforms.
• Advanced Python, PySpark and SQL skills.
• Strong understanding of converting Ab Initio transformations/reformat logic into scalable PySpark solutions.
• Experience building and optimising large-scale ETL/data pipeline solutions in AWS.
• Excellent Spark performance tuning capabilities, including troubleshooting data skew, partitioning, memory and load-related issues.
• Strong data warehousing and data modelling knowledge
• Strong comminication skills.

Nice to Have

• Experience with Claude Code or AI-assisted code conversion tools.
• Amazon MWAA (Airflow) experience.
• AWS Certifications.
• Experience working on large-scale legacy modernisation programmes.