Senior Data Engineer

CreateFuture; Manchester; Hybrid; GBP 70,000 - 110,000; Full time

Benefits offered by this job

Flexible working

Hybrid/Remote options

Office hubs in Edinburgh, Leeds, Leeds

Job summary

CreateFuture in the UK is hiring a Senior Data Engineer to join our Data Practice. You’ll deliver data solutions for major operators, handling sub-second odds feeds, regulatory reporting, and large-scale cloud migrations from on-prem to cloud-native AWS architectures.

You’ll design low-latency pipelines using AWS services and streaming tech, validate data quality, and collaborate with cross-functional teams. Hybrid/remote work available across our UK hubs, with mentoring responsibilities.

Qualifications

• Extensive experience delivering data migrations to cloud platforms.
• Strong knowledge of real-time data pipelines and streaming.
• Experience with AWS data services and cloud-native architectures.
• Familiarity with iGaming data domains and regulatory reporting.

Responsibilities

• Deliver large-scale data migrations to cloud-native AWS architectures with Lakehouse implementations.
• Design, build and optimise low-latency data pipelines for real-time betting events.
• Implement data quality, validation, reconciliation and audit controls across data domains.
• Develop streaming solutions using AWS Kinesis, Kafka or MSK for real-time feeds.
• Utilise AWS services such as S3, Glue, Lambda, Athena, Redshift, Lake Formation and MWAA.
• Support regulatory reporting and ensure data pipelines are auditable and compliant.
• Collaborate with architects and cross-functional teams to design scalable data platforms.
• Mentor junior engineers and contribute to solution design sessions and client workshops.

Skills

Python

SQL

Spark

dbt

Airflow

Tools

Terraform

AWS CDK

Databricks

Snowflake

AWS Kinesis

Great Expectations

Job description

Who we are

CreateFuture is fast becoming the UK's most recognisable digital consultancy, with years of experience building digital products and services for major organisations whilst putting our people first. We have offices in the centre of Edinburgh, Leeds, Manchester, and London as well as remote employees located throughout the country.

Who we are

CreateFuture is fast becoming the UK's most recognisable digital consultancy, with years of experience building digital products and services for major organisations whilst putting our people first. We have offices in the centre of Edinburgh, Leeds, Manchester, and London as well as remote employees located throughout the country. We are a team of creators - whether that's code, project plans, go to market strategies, culture initiatives, marketing campaigns, large language models or people policies. And together, with our clients, we create the future. This has seen us collaborate and partner across a multitude of industries and sectors, with the likes of PayPal, adidas, Natwest, FanDuel and Money Saving Expert, to name just a few. Our reputation as a partner determined to deliver high-quality, robust and thoughtful products has enabled us to scale to over 500 people in the last couple of years, and it is our amazing people - along with the safe, supportive and friendly culture we have built - that makes CreateFuture a great place to work. Don't just take our word for it though, we have been recognised by Best Workplaces UK multiple years in a row - across a number of categories - and our employee exit rate is astonishingly low.

Join us on our journey… Let's create something awesome, together, today.

About the role and team:

We're looking for an experienced Senior Data Engineer to join our growing Data Practice. You'll work as part of a multidisciplinary team delivering data solutions for some of the world's largest betting and gaming operators — organisations processing millions of events per day across sportsbook, casino, and exchange platforms. This is a role for someone who understands the pressures of the iGaming data landscape: sub-second odds feeds, high-stakes data integrity requirements, complex regulatory reporting, and large-scale platform migrations from legacy on-premises estates to cloud-native architectures on AWS.

What you'll be doing

• Deliver large-scale data migrations from on-premises or legacy platforms to cloud-native AWS architectures, including Lakehouse implementations using Databricks or Snowflake.
• Design, build, and optimise low-latency data pipelines that process real-time betting events, odds updates, and market data at scale.
• Implement robust data quality, validation, reconciliation, and audit controls across betting, trading, and customer data domains.
• Develop streaming and micro-batch data solutions using AWS Kinesis, Apache Kafka, or Amazon MSK to support real-time sportsbook and in-play data feeds.
• Build and maintain data solutions using AWS services including S3, Glue, Lambda, Athena, Redshift, Lake Formation, Step Functions, and MWAA.
• Support regulatory and compliance reporting by ensuring data pipelines are reliable, traceable, and meet audit and lineage requirements for the UK Gambling Commission and other international regulators.
• Collaborate with architects, platform engineers, and cross-functional teams to design scalable, resilient, and high-performing data platforms.
• Champion engineering best practices through high-quality code, automated testing, documentation, and peer code reviews, while mentoring junior engineers where appropriate.
• Contribute technical expertise during solution design sessions, architecture discussions, and client workshops, providing input into delivery approaches and technical decisions.

We'd love to talk to you if...

• You've worked in or closely with sportsbook, exchange, or casino platforms — and understand the data models behind markets, selections, bets, and settlements
• You've built or maintained low-latency data feeds — odds ingestion, in-play event streams, price change feeds — where milliseconds matter
• You understand the data integrity challenges specific to betting: reconciliation between trading and settlement systems, detection of data drift, and auditability for regulatory purposes
• You've delivered or contributed to large-scale data migrations — moving high-volume transactional or event data from legacy estates to the cloud without losing fidelity or uptime

You know the iGaming data landscape

• You've worked in or closely with sportsbook, exchange, or casino platforms — and understand the data models behind markets, selections, bets, and settlements
• You've built or maintained low-latency data feeds — odds ingestion, in-play event streams, price change feeds — where milliseconds matter
• You understand the data integrity challenges specific to betting: reconciliation between trading and settlement systems, detection of data drift, and auditability for regulatory purposes
• You've delivered or contributed to large-scale data migrations — moving high-volume transactional or event data from legacy estates to the cloud without losing fidelity or uptime

You're strong on AWS data engineering

• AWS is your primary cloud — you're confident across Glue, Lambda, Athena, S3, Redshift, RDS, Kinesis, MSK, and Step Functions
• You've built real-time or near-real-time pipelines using streaming frameworks — Kinesis Data Streams, Kafka, Flink, or Spark Streaming
• You understand AWS cost models and how to optimise high-throughput workloads — Spot compute, reserved capacity, and right-sizing for event-driven architectures
• AWS certifications are a bonus

You know your tools

• Solid experience with Python, SQL, and Spark — and pipeline tools such as dbt or Airflow
• Comfortable with data quality frameworks and testing — Great Expectations, dbt tests, or custom validation layers
• Experience with infrastructure-as-code (Terraform, AWS CDK) is a plus
• Familiar with broader cloud platforms — Azure (Data Factory, Synapse) or GCP (Dataflow, BigQuery) — is beneficial but not essential

Tech snapshot

Primary cloud

AWS — Glue, Lambda, Athena, S3, Redshift, RDS, Kinesis, MSK, Lake Formation, Step Functions, MWAA

Streaming

AWS Kinesis, Apache Kafka / MSK, Apache Flink, Spark Streaming

Languages

Python, SQL, Spark

Pipeline tools

dbt, Airflow / AWS MWAA, AWS Glue

Platforms

Databricks, Snowflake, Amazon Redshift

Data quality

Great Expectations, dbt tests, custom reconciliation frameworks

IaC / DevOps

Terraform, AWS CDK (desirable)

What we’ll offer you

We trust people to do their best work. That means flexibility over rigid rules, impact over activity, and real investment in your growth both professionally and personally. You'll be part of a supportive, and friendly culture, surrounded by smart, curious people who care deeply about what they do. We offer flexible working, including hybrid and remote options. Our office hubs are located in Edinburgh, Leeds, Manchester, London and Bulgaria, with occasional travel to client sites or CreateFuture offices when needed. We trust you to manage your time balancing collaboration with client time and focused work. What matters is the impact you have, not how busy you look.

Our hiring process

We try to keep our hiring process clear, fair and respectful of your time. We aim to get back to everyone who applies and we will be upfront about where you are in the process. It usually looks like this:

• Call with our Talent Acquisition Team
• Role specific capability interview

Inclusion at CreateFuture

We believe diverse teams build better workplaces and better products. We want CreateFuture to be a place where people feel able to be themselves and do their best work. If you need any adjustments or support during the application process, just. We will do what we can to help.

We look forward to your application!