Lead Data Engineer

New Day; London; On-site; GBP 70,000 - 90,000; Full time

Job summary

A leading financial services provider in the UK seeks a Senior Data Engineer to lead a team of engineers. Responsibilities include defining the technical vision, managing data solutions, and collaborating with various stakeholders. The ideal candidate has extensive experience in data engineering, strong skills in SQL and Python, and hands-on experience with AWS services like Snowflake. This role offers opportunities for leadership and innovation in a fast-paced environment.

Qualifications

• Proven experience as a Senior or Lead Data Engineer with complex data pipelines.
• Strong proficiency in Python and SQL.
• Hands-on experience with AWS services and Snowflake.

Responsibilities

• Lead and mentor a team of 3-6 data engineers.
• Define the technical vision for data solutions.
• Collaborate with stakeholders for successful delivery.
• Develop and maintain scalable data pipelines.

Skills

Data Engineering Experience

Technical Expertise in Python

Proficiency in SQL

AWS Services

Experience with Snowflake

Big Data Processing with Spark

Workflow Orchestration with Airflow

DBT (Data Build Tool)

Proficient with Git

Job description

What you will deliver

• Lead and Mentor: Lead, manage, and mentor a team of 3-6 data engineers, fostering a collaborative and high-performing environment.
• Technical Vision & Ownership: Own and define the technical vision, providing architectural guidance and best practices for data modelling, ETL/ELT processes, data warehousing, and data lake solutions.
• Collaboration & Stakeholder Management: Work closely with Product Owners, Business Analysts, and other stakeholders to understand requirements, translate them into technical specifications, and ensure successful delivery of data solutions.
• Hands-on Development: Be a hands-on contributor to the design, development, and maintenance of scalable data pipelines and platforms, leveraging a modern data stack.
• Data Quality & Testing: Implement and champion robust testing strategies for data pipelines, including unit testing, integration testing, and data quality checks, to ensure accuracy, reliability, and completeness of data.

Your Skills and Experience

Essential

• Extensive Data Engineering Experience: Proven experience as a Senior or Lead Data Engineer, with a strong track record of designing, building, and maintaining complex data pipelines and platforms.

• Technical Expertise:
  
  • Languages: Strong proficiency in Python and SQL.
  • Cloud Platform: Hands-on experience with AWS services (EMR, Athena, Lambda, S3, Glue, Step Functions).
  • Data Warehousing: In-depth experience with Snowflake.
  • Big Data Processing: Experience with Spark and familiarity with Scala (even if limited).
  • Orchestration: Solid experience with Airflow for workflow orchestration.
  • Data Transformation: Hands-on experience with DBT (Data Build Tool).
  • Version Control: Proficient with Git and GitHub, including experience with GitHub Actions for CI/CD.

• Leadership & Management:
  
  • Demonstrated experience in leading, mentoring, and managing a team of data engineers.
  • Ability to motivate and inspire a team, fostering a culture of continuous learning and excellence.
  • Strong communication and interpersonal skills, with the ability to effectively collaborate with technical and non-technical stakeholders.

Desirable

• Real-time data processing
• Data governance and data quality frameworks
• Data operations and observability

We work with Textio to make our job design and hiring inclusive.

Permanent