Lead Data Engineer

F5; London; Hybrid; GBP 100,000 - 120,000; Full time

Job summary

An innovative FinTech company is seeking a Lead AWS Data Engineer to spearhead the design and implementation of data solutions. In this hybrid role, you will lead a talented team, leveraging your extensive AWS expertise to enhance data quality and support regulatory compliance for financial institutions. This position offers the opportunity to work with cutting-edge technologies like Glue, Athena, and Kinesis, while promoting best practices in data engineering. Join a collaborative environment that values personal development and team success, and enjoy a comprehensive benefits package including private healthcare and gym contributions.

Benefits

25 days holiday + bank holidays + birthday off

Private healthcare & medical cashback plan

Monthly gym membership contribution

Annual personal development allowance

Regular team socials

Qualifications

• 7+ years of hands-on experience in data engineering and ETL architecture.
• Deep expertise in AWS Data Services and strong programming skills in Python.

Responsibilities

• Lead and mentor a team of Data Engineers, driving technical growth.
• Design and optimise scalable data pipelines using AWS services.

Skills

Leadership in Data Engineering

AWS Data Services (Glue, Athena, Redshift)

Python Programming

PySpark

SQL (Spark-SQL, MySQL, Presto SQL)

NoSQL Databases (DynamoDB, MongoDB)

Infrastructure-as-Code (Terraform, CloudFormation)

ETL Architecture

Job description

Job Description

Lead AWS Data Engineer

City ofLondon (Hybrid: Around 2-3 office days per week)

Salary: Up to £120,000

Industry: FinTech / RegTech

A 200 employee data and technology company specialising in Regulatory Technology (RegTech) are looking for a Lead AWS Data Engineer. They develop products that enhance data quality and help financial institutions navigate regulatory compliance.

The Role

The Lead Data Engineer will provide technical leadership to a small team of Data Engineers (onshore & offshore), driving the design, implementation, and optimisation of scalable data solutions in AWS.

Key Responsibilities

• Lead and mentor a team of onshore and offshore Data Engineers, driving technical growth and ensuring effective collaboration.
• Design, develop, and optimise scalable data pipelines and infrastructure using AWS (Glue, Athena, Redshift, Kinesis, Step Functions, Lake Formation).
• Utilise PySpark for distributed data processing, ETL, SQL querying, and real-time data streaming.
• Architect and implement robust data solutions for analytics, reporting, machine learning, and data science initiatives.
• Establish and enforce best practices in data engineering, coding standards, and architecture guidelines.
• Manage team tasks and delivery using Agile methodologies, collaborating closely with product owners and scrum masters.

Skills and Experience

• 2+ years of leadership experience in data engineering, including mentoring and team management.
• 7+ years of hands-on experience in data engineering and ETL architecture.
• Deep expertise in AWS Data Services, including Glue, Athena, Redshift, Kinesis, Step Functions, and Lake Formation.
• Strong programming skills in Python and PySpark for data processing and automation.
• Extensive SQL experience (Spark-SQL, MySQL, Presto SQL) and familiarity with NoSQL databases (DynamoDB, MongoDB, etc.).
• Proficiency in Infrastructure-as-Code (Terraform, CloudFormation) for automating AWS data workflows.
• Experience designing, implementing, and optimising ETL pipelines, including real-time data processing and streaming architectures.

25 days holiday + bank holidays + birthday off.

Private healthcare & medical cashback plan.

Monthly gym membership contribution.

Annual personal development allowance.

Regular team socials in a collaborative environment.