Lead Data Engineer

Betfred Group Holdings Limited; Manchester; Hybrid; GBP 85,000 - 105,000; Full time

Benefits offered by this job

Pension

Discretionary bonus

Generous annual leave

Online GP

Employee assistance

Maternity & Paternity Pay

Long service recognition

Wagestream wage advance

Job summary

Betfred Group Holdings Limited in Manchester is seeking a Lead Data Engineer to set the architectural direction for our data platform, driving the migration from SQL Server to an AWS lakehouse and the real‑time streaming architecture that underpins analytics.

You will mentor engineers, own IaC and CI/CD standards, and collaborate with data scientists and analysts to deploy ML solutions while controlling cost and security.

Qualifications

• Strong hands-on AWS data services experience.
• Advanced Python and SQL, with PySpark for large-scale processing.
• Experience designing streaming architectures with Kafka or MSK.
• Experience with Iceberg or Delta Lake for open table formats.
• Terraform for infrastructure-as-code and CI/CD.
• Proven leadership mentoring engineers and setting standards.
• Ability to communicate with senior leadership.
• Familiar with Agile methodologies.
• Desirable: Flink, on-prem to cloud migrations, regulated industry awareness.

Responsibilities

• Own architecture, standards and design patterns for the data platform (batch and streaming on AWS).
• Lead the Data Warehouse migration from SQL Server to AWS; modelling and cutover strategy.
• Design real-time data architecture using MSK/Kafka and PySpark; exactly-once semantics.
• Define data quality, observability and governance frameworks.
• Govern lakehouse and data warehouse design for scalability and cost.
• Own IaC (Terraform) and CI/CD standards.
• Monitor AWS spend and optimize performance vs cost.
• Escalation point for complex data issues; drive root-cause fixes.
• Collaborate with data scientists/analysts on ML deployments.
• Mentor engineers; participate in hiring assessments.
• Represent data engineering in cross-functional planning; translate tech trade-offs.
• Embed data security/privacy with Security and DP teams.

Skills

Python

SQL

PySpark

Data modeling

Streaming

Tools

AWS Glue

AWS Lambda

AWS S3

Redshift

Kafka (MSK)

Terraform

Job description

About Us

Our mission is to dominate the betting and gaming industry on a global scale and we need the very best Tech talent to help us achieve this.

With our proprietary platform powering everything we do, it's an exciting time to join us. We're pioneering new products and driving more advanced, creative technologies. The result? Unrivalled experiences for millions of customers worldwide.

Betfred's Technology department is driven by innovation, and you'll be at the heart of unlocking our platform's potential. So, if you want to help shape the future of betting and gaming, it's time to join us.

Job Purpose

This Lead Data Engineer will be the senior technical authority for our data platform, setting the architectural direction and engineering standards for the team. This hands‑on role will focus on leading the technical delivery of our migration from SQL Server to an AWS lakehouse, and designing the real‑time streaming architecture that underpins automation, business intelligence, and analytics, while mentoring engineers across the team.

Job Duties

Own the technical architecture, standards and design patterns for the data platform, covering batch and streaming pipelines built on AWS Glue, AWS Lambda, AWS S3 and Redshift.

Lead the technical delivery of the Data Warehouse migration from SQL Server to AWS, including target modelling, cutover approach, reconciliation strategy and technical risk management.

Design and build the real‑time data architecture using Amazon MSK / Apache Kafka and PySpark Structured Streaming (evaluating Apache Flink where it is the better fit), including event schema governance, exactly‑once semantics and stream‑to‑lakehouse integration.

Define and embed data quality, validation and observability frameworks, and hold the team to them through design and code review.

Govern the lakehouse and data warehouse design, ensuring consistency, scalability and sensible cost characteristics across the platform.

Own platform infrastructure‑as‑code (Terraform) and CI/CD standards, so environments and pipelines are reproducible and reviewable.

Monitor and optimise AWS platform spend, making the cost and performance trade‑offs explicit when proposing architecture.

Act as the technical escalation point for complex data issues and performance bottlenecks, leading root‑cause analysis and driving permanent fixes.

Collaborate with data scientists, analysts and other engineering leads to support the deployment of machine learning and advanced analytical solutions.

Mentor and technically coach data engineers through pairing, review and design sessions, and contribute to technical assessment during hiring.

Represent data engineering in cross‑functional architecture, planning and prioritisation discussions, translating technical trade‑offs for senior technical and non‑technical stakeholders.

Embed data security and privacy by design, working with Information Security and our Data Protection team on how customer data is modelled, accessed and retained.

Knowledge, Skills and Experience

Essentials:

Deep, hands‑on expertise across core AWS data services such as AWS Glue, AWS Lambda, AWS S3, AWS Redshift, and Amazon EMR, with the ability to make and defend architectural decisions.

Advanced proficiency in Python and SQL, including PySpark for large‑scale batch and streaming processing, and strong command of data modelling and data warehousing concepts.

Production experience designing and operating streaming architectures with Apache Kafka or Amazon MSK, including stream processing with Spark Structured Streaming.

Hands‑on experience with open table formats, such as Apache Iceberg or Delta Lake, including partitioning, schema evolution and table maintenance at scale.

Experience with pipeline orchestration and infrastructure‑as‑code using Terraform.

Experience acting as technical lead on a significant data platform or migration programme, setting standards others follow.

Track record of mentoring and developing engineers through review, pairing and design guidance.

Ability to diagnose and resolve complex, ambiguous data issues, and to teach others to do the same.

Ability to effectively communicate with both technical and non‑technical stakeholders, including senior leadership.

Experience with data security and privacy best practices, and how to build them into platform design.

Experience with Agile development methodologies.

Desirable:

Experience with Apache Flink for stateful stream processing would be desirable.

Experience in migrating data from on‑premises systems, such as SQL Server, to cloud environments would be desirable.

Experience in a regulated industry, and awareness of how data platform design intersects with regulatory, compliance and responsible gambling obligations, would be an advantage.

What is in it for me?

We offer a variety of competitive benefits, some of which vary depending on the role you’re recruited to. Some of what you can expect in this role includes:

A competitive rate of pay and pension contribution ( £85,000 -£105,000)

Generous discretionary bonus schemes, incentives and competitions

An annual leave entitlement that increases with length of service

Access to an online GP 24/7, 365 days a year for you and your immediate family.

Employee wellbeing support through our Employee Assistance Programme

Enhanced Maternity & Paternity Pay

Long Service Recognition

Access to a pay day savings scheme, financial coach and up to 40% of your earned wage ahead of payday, through Wagestream.