Data Engineering Lead

ZipRecruiter; London; On-site; GBP 60,000 - 100,000; Full time

Job summary

An established industry player is seeking a Lead Data Engineer to spearhead an expanding data engineering team in London. This pivotal role involves overseeing the creation of a next-gen data lake using AWS and Dremio, while ensuring the integrity and enhancement of data through effective monitoring and cleaning processes. As a leader, you will guide a talented team, set best practices, and maintain communication with C-suite stakeholders, aligning engineering efforts with the broader architectural vision. If you are passionate about leveraging data to drive business insights and thrive in a collaborative environment, this opportunity is perfect for you.

Qualifications

• 7-8 Jahre Erfahrung in Python, insbesondere in Datenanalyse und Automatisierung.
• Starke SQL-Kenntnisse, insbesondere in Bezug auf ETL-Prozesse.

Responsibilities

• Leitung eines Teams von Data Engineers und Sicherstellung der Projektlieferung.
• Entwicklung robuster ETL-Pipelines zur Integration in das Unternehmen.

Skills

Python programming

SQL

AWS (S3, Glue, Lambda, Step Functions)

PowerBI

Dremio

Data Mesh Architecture

Team leadership

Communication skills

Tools

AWS

PowerBI

Dremio

Job description

Job Description

Lead Data Engineer

Location: London

A Lead Data Engineer is sought after by a leading workplace pensions provider to run their expanding data engineering team in their London City office. The team has recently built out a next-gen data lake hosted in AWS which makes use of Dremio, Data Mesh architecture, and a wide range of AWS tools (S3, Glue, Step Functions, etc.).

With the infrastructure of this new Data Lake in place, there is now a focus on enhancing the data stored within (via monitoring, cleaning) and a requirement to design, build, and implement robust ETL pipelines for effective integration with the wider business.

As a Lead Engineer, you will also be responsible for overseeing a small team of Data Engineers, setting and driving best practices and standards, ensuring project delivery is kept on track, and regularly engaging with stakeholders up to C-suite for feedback/updates. This will also involve working directly alongside the Head of Data Platforms to ensure ongoing engineering work is aligned to overall architecture and roadmap, as well as future project planning.

Required Skills

1. 7/8 years’ experience in Python programming, particularly around data analysis, automation, and the building of data pipelines (as well as an understanding of data warehousing concepts)
2. Strong SQL skills, especially in relation to ETL processes and data management (querying, cleaning, storing)
3. Experience working with and deploying to AWS cloud (having worked with any/all of S3, Glue, Lambda, Step Functions, etc.)
4. Usage of PowerBI for data visualization
5. Experience of both Dremio and Data Mesh Architecture
6. Prior experience of leading software engineering teams (including line management)
7. Clear, confident communications with previous experience of working directly with business users