Lead Data Engineer

New Day; City Of London; On-site; GBP 70,000 - 90,000; Full time

Job summary

A data solutions company is seeking a Lead Data Engineer to manage a small team while defining the technical vision for data solutions. The ideal candidate will have extensive experience in data engineering, proficiency in Python and SQL, and strong skills in AWS and data warehousing. This permanent position offers the opportunity to work in a collaborative environment, driving the development of scalable data pipelines and fostering a culture of learning and excellence.

Qualifications

• Proven experience as a Senior or Lead Data Engineer with a strong track record.
• Experience leading and managing a team of data engineers.
• Ability to effectively collaborate with technical and non-technical stakeholders.

Responsibilities

• Lead, manage, and mentor a team of 3-6 data engineers.
• Define the technical vision and best practices for data engineering.
• Collaborate with stakeholders to translate requirements into technical specifications.
• Contribute to the design and development of scalable data pipelines.

Skills

Data engineering experience

Proficiency in Python

Proficiency in SQL

Hands-on experience with AWS

In-depth experience with Snowflake

Experience with Spark

Solid experience with Airflow

Proficient with Git and GitHub

Job description

What you will deliver

• Lead and Mentor: Lead, manage, and mentor a team of 3-6 data engineers, fostering a collaborative and high-performing environment.
• Technical Vision & Ownership: Own and define the technical vision, providing architectural guidance and best practices for data modelling, ETL/ELT processes, data warehousing, and data lake solutions.
• Collaboration & Stakeholder Management: Work closely with Product Owners, Business Analysts, and other stakeholders to understand requirements, translate them into technical specifications, and ensure successful delivery of data solutions.
• Hands-on Development: Be a hands-on contributor to the design, development, and maintenance of scalable data pipelines and platforms, leveraging a modern data stack.
• Data Quality & Testing: Implement and champion robust testing strategies for data pipelines, including unit testing, integration testing, and data quality checks, to ensure accuracy, reliability, and completeness of data.

Your Skills and Experience

Essential

• Extensive Data Engineering Experience: Proven experience as a Senior or Lead Data Engineer, with a strong track record of designing, building, and maintaining complex data pipelines and platforms.

• Technical Expertise:
  
  • Languages: Strong proficiency in Python and SQL.
  • Cloud Platform: Hands-on experience with AWS services (EMR, Athena, Lambda, S3, Glue, Step Functions).
  • Data Warehousing: In-depth experience with Snowflake.
  • Big Data Processing: Experience with Spark and familiarity with Scala (even if limited).
  • Orchestration: Solid experience with Airflow for workflow orchestration.
  • Data Transformation: Hands-on experience with DBT (Data Build Tool).
  • Version Control: Proficient with Git and GitHub, including experience with GitHub Actions for CI/CD.

• Leadership & Management:
  
  • Demonstrated experience in leading, mentoring, and managing a team of data engineers.
  • Ability to motivate and inspire a team, fostering a culture of continuous learning and excellence.
  • Strong communication and interpersonal skills, with the ability to effectively collaborate with technical and non-technical stakeholders.

Desirable

• Real-time data processing
• Data governance and data quality frameworks
• Data operations and observability

We work with Textio to make our job design and hiring inclusive.

Status: Permanent