Lead Data Engineer

ITV; Greater London; Hybrid; GBP 60,000 - 100,000; Full time

Job summary

An established industry player is seeking a Lead Data Engineer to spearhead innovative data solutions and lead a talented team. This exciting role involves designing and optimizing data pipelines using cutting-edge technologies like Databricks, while collaborating with cross-functional teams to deliver impactful data products. If you have a passion for data, leadership, and driving continuous improvement, this opportunity offers a chance to shape the future of data engineering. Join a forward-thinking organization that values innovation and teamwork, and make a significant impact on the data landscape.

Qualifications

• Experience in leading a team of data engineers.
• Extensive knowledge in data engineering and cloud platforms.
• Ability to write production-grade code with automated testing.

Responsibilities

• Lead a team of data engineers, driving productivity and growth.
• Design and maintain scalable data pipelines using Databricks.
• Champion data governance and ensure compliance with regulations.

Skills

Leadership

Data Engineering

Databricks

Python (PySpark)

SQL

Cloud Platforms (AWS, Azure, GCP)

CI/CD (Github Actions, Jenkins)

Data Governance

Problem-Solving

Collaboration

Tools

Databricks

Apache Spark

Tableau

AWS QuickSight

Looker

ThoughtSpot

Job description

Workplace: London White City - expectation is 2 days in the office.

The role

As a Lead Data Engineer, both directly and via your team of data engineers, you will be responsible for developing, maintaining, and optimising our data pipeline infrastructure using our proprietary central data platform, which is based on Databricks. You will collaborate with cross-functional teams to design and implement scalable data solutions, ensuring efficient data ingestion, transformation, storage, and analysis.

This role offers the chance to lead impactful projects, work with cutting-edge technologies, and shape the future of data engineering at ITV. If you’re passionate about data, leadership, and innovation, we’d love to hear from you!

Key responsibilities

• Lead and Inspire: Take charge of a talented team of data engineers, providing technical leadership, mentorship, and fostering a collaborative environment. Drive productivity, set clear objectives, and support professional growth while contributing to performance reviews and recruitment
  
• Design & Build Scalable Data Solutions: Own the end-to-end design, development, and maintenance of cutting-edge data pipelines and infrastructure using Databricks. Innovate with new approaches, ensuring efficient data ingestion, transformation, and storage at scale
  
• Collaborate Across Teams: Work closely with cross-functional teams, including data scientists, analysts, and commercial stakeholders, to define data product requirements and deliver impactful data visualisation solutions
  
• Performance & Quality Assurance: Monitor and optimise data pipelines for peak performance and reliability. Implement automated quality processes to ensure data accuracy and report outcomes through advanced observability tools
  
• Governance & Security: Champion data governance practices, ensuring compliance with data privacy and security regulations. Establish access controls, encryption, and retention policies to safeguard sensitive information
  
• Continuous Improvement: Stay ahead of industry trends, identifying opportunities for innovation and automation. Engage with the ITV Symphony team to drive feature enhancements and contribute to the broader data engineering community.

Qualifications and Skills

• Experience in leading a small team of data engineers
• Extensive knowledge as a Data Engineer
• Demonstrable success in designing and building Databricks/Snowflake/GCP Big Data/Hadoop (Cloudera/HortonWorks)/Spark data products
• Excellent problem-solving, analytical thinking, and troubleshooting skills
• Strong communication and collaboration abilities, with the ability to work effectively in a team environment
• Strong programming skills in languages such as Python (PySpark preferred), Scala, or SQL
• Experience in owning, designing and implementing data pipelines ingesting enterprise levels of data volume. In-depth knowledge of data engineering concepts, data integration, and ETL processes
• Ability to write production-grade code including automated testing
• Experience in deploying code via CI/CD platforms (e.g. Github Actions, Jenkins)
• Proficiency in working with distributed computing frameworks, such as Apache Spark
• Experience with cloud platforms (e.g., AWS, Azure, Google Cloud) and associated services (e.g., S3, Redshift, BigQuery)
• Familiarity with data modelling, database systems, and SQL optimisation techniques

Other things we’re looking for (key criteria)

• Knowledge of the UK broadcast industry & broadcast/OTT advertising market
• Knowledge of digital marketing and advertising industry
• Familiar with data design techniques for classification and regression as well as knowledge of A/B testing, experimental design, and general statistical modelling
• Familiarity with architecture disciplines such as Data Mesh Architecture, Data Architecture, BI Architecture and Enterprise Architecture
• Good understanding of data governance, data privacy, and security principles.
• Knowledge of tools across other data management capabilities – data engineering, data governance, data quality, metadata management and master data management
• Knowledge of designing and building out data visualisation solutions based on technology such as: Tableau, AWS QuickSight, Looker, ThoughtSpot