Senior Data Platform Lead – AWS & Real-Time Analytics

Betfred Group Holdings Limited; Manchester; Hybrid; GBP 85,000 - 105,000; Full time

Benefits offered by this job

Pension

Discretionary bonus

Generous annual leave

Online GP

Employee assistance

Maternity & Paternity Pay

Long service recognition

Wagestream wage advance

Job summary

Betfred Group Holdings Limited in Manchester is seeking a Lead Data Engineer to set the architectural direction for our data platform, driving the migration from SQL Server to an AWS lakehouse and the real‑time streaming architecture that underpins analytics.

You will mentor engineers, own IaC and CI/CD standards, and collaborate with data scientists and analysts to deploy ML solutions while controlling cost and security.

Qualifications

• Strong hands-on AWS data services experience.
• Advanced Python and SQL, with PySpark for large-scale processing.
• Experience designing streaming architectures with Kafka or MSK.
• Experience with Iceberg or Delta Lake for open table formats.
• Terraform for infrastructure-as-code and CI/CD.
• Proven leadership mentoring engineers and setting standards.
• Ability to communicate with senior leadership.
• Familiar with Agile methodologies.
• Desirable: Flink, on-prem to cloud migrations, regulated industry awareness.

Responsibilities

• Own architecture, standards and design patterns for the data platform (batch and streaming on AWS).
• Lead the Data Warehouse migration from SQL Server to AWS; modelling and cutover strategy.
• Design real-time data architecture using MSK/Kafka and PySpark; exactly-once semantics.
• Define data quality, observability and governance frameworks.
• Govern lakehouse and data warehouse design for scalability and cost.
• Own IaC (Terraform) and CI/CD standards.
• Monitor AWS spend and optimize performance vs cost.
• Escalation point for complex data issues; drive root-cause fixes.
• Collaborate with data scientists/analysts on ML deployments.
• Mentor engineers; participate in hiring assessments.
• Represent data engineering in cross-functional planning; translate tech trade-offs.
• Embed data security/privacy with Security and DP teams.

Skills

Python

SQL

PySpark

Data modeling

Streaming

Tools

AWS Glue

AWS Lambda

AWS S3

Redshift

Kafka (MSK)

Terraform

Job description

Betfred Group Holdings Limited in Manchester is seeking a Lead Data Engineer to set the architectural direction for our data platform, driving the migration from SQL Server to an AWS lakehouse and the real‑time streaming architecture that underpins analytics.

You will mentor engineers, own IaC and CI/CD standards, and collaborate with data scientists and analysts to deploy ML solutions while controlling cost and security.