Data Engineering Lead

Fairmont Recruitment Technology; Greater London; On-site; GBP 90,000 - 130,000; Full time

Job summary

Fairmont Recruitment Technology is seeking a Data Engineering Lead to provide senior technical leadership across a large-scale AWS data platform for a major organisation. This role blends hands-on engineering with team leadership, platform ownership and architectural decision-making.

You will mentor engineers, define engineering standards, collaborate with architecture, security and QA teams, and help translate business requirements into scalable data solutions while driving DataOps practices

Qualifications

• Advanced SQL skills and optimisation across very large datasets.
• Experience building batch and streaming data pipelines.
• Knowledge of data warehouses, data lakes and modern storage patterns.
• Experience with Spark or similar distributed processing tech.
• Strong CI/CD, DevOps and DataOps practices.
• Setting engineering standards and improving code quality.
• Previous technical leadership or mentoring experience.
• Strong stakeholder management and communication skills.
• Experience with large-scale data and complex operational environments.
• Experience integrating legacy systems, fixed-width files and REST APIs.
• Experience working with external technology and delivery partners.

Responsibilities

• Lead the development, operation and improvement of the AWS data platform.
• Oversee delivery of complex batch and streaming data pipelines.
• Make architectural decisions around data models, storage and tooling.
• Improve platform resilience, scalability, performance and availability.
• Set standards across coding, testing, monitoring, documentation and DataOps.
• Mentor and support data engineers at different levels.
• Act as the senior escalation point for complex data engineering issues.
• Work with Architecture, Information Security and QA teams.
• Collaborate with external partners on integrations and platform development.
• Translate business requirements into practical technical solutions.
• Prioritise engineering work and support the organisation’s data strategy.
• Ensure operational controls and monitoring are in place.

Skills

Advanced SQL

Batch processing

Streaming data pipelines

Data warehouses

Data lakes

Spark

CI/CD

DevOps

DataOps

Coding standards

Mentoring

Stakeholder management

Large datasets

Operational environments

Legacy integrations

REST APIs

Partner collaboration

Education

Tools

AWS Redshift

Amazon S3

AWS Glue

AWS Lambda

EC2

Kinesis Firehose

Python

SQL

Bash

Job description

We are recruiting a Data Engineering Lead to provide senior technical leadership across a large-scale AWS data platform supporting a major organisation.

This role combines hands-on technical expertise with team leadership, platform ownership and architectural decision-making. You will act as the senior authority on data engineering, helping ensure the platform remains reliable, scalable and capable of supporting increasingly complex data requirements.

You will work closely with senior data and architecture leaders while mentoring engineers and improving engineering standards across the wider organisation.

Key responsibilities

• Lead the development, operation and improvement of the AWS data platform
• Oversee the delivery of complex batch and streaming data pipelines
• Make technical decisions around architecture, data models, storage and engineering tooling
• Improve platform resilience, scalability, performance and availability
• Set standards across coding, testing, monitoring, documentation and DataOps
• Mentor and support data engineers at different levels
• Act as the senior escalation point for complex data engineering issues
• Work closely with Architecture, Information Security and Quality Assurance teams
• Collaborate with external partners on integrations and platform development
• Translate business requirements into practical technical solutions
• Help prioritise engineering work and support the organisation’s wider data strategy
• Ensure suitable operational controls and monitoring are in place

Technical environment

The platform uses a broad range of AWS services, including:

• Redshift
• S3
• Glue
• Lambda
• EC2
• Kinesis Firehose

The engineering environment also makes extensive use of Python, SQL and Bash, with data arriving through both legacy and modern source systems.

• Advanced SQL skills, including optimisation across very large datasets
• Experience building batch and streaming data pipelines
• Knowledge of data warehouses, data lakes and modern data storage patterns
• Experience with distributed processing technologies such as Spark
• Strong understanding of CI/CD, DevOps and DataOps practices
• Experience setting engineering standards and improving code quality
• Previous technical leadership or mentoring experience
• Strong stakeholder management and communication skills
• Experience working with exceptionally large datasets
• Exposure to complex operational environments
• Experience integrating legacy systems, fixed-width files and REST APIs
• Experience working with external technology and delivery partners