Senior Data Engineer

WGSN; Greater London; On-site; GBP 70,000 - 90,000; Full time

Job summary

A leading forecasting agency is looking for a Senior Data Engineer to enhance their data architecture and support high-performance data pipelines in London. The ideal candidate will have over 5 years of data engineering experience, expert-level SQL skills, and proficiency with Snowflake and Databricks. You will work closely with data scientists and engineers to optimize data flows and ensure data quality. The company offers a flexible working environment with a strong focus on employee well-being and development.

Benefits

25 days of holiday

Pension and life assurance

Flexible benefits platform

Qualifications

• Experience building scalable data architectures.
• Familiarity with cloud environments like AWS or GCP.
• Ability to mentor junior engineers.

Responsibilities

• Design and maintain scalable data architectures.
• Optimize SQL queries and improve performance.
• Build robust ETL/ELT pipelines.

Skills

SQL

Data modelling

Python

Data pipeline orchestration

Data validation frameworks

Cloud platforms (AWS or GCP)

Communication

Problem-solving

Education

5+ years of experience in data engineering

Tools

Snowflake

Databricks

Airflow

Docker

Job description

The role

We are looking to hire a Senior Data Engineer to join our Data team in London.

This is an office-based role out of our London office.

Working at WGSN

Together, we create tomorrow

A career with WGSN is fast-paced, exciting and full of opportunities to grow and develop. We're a team of consumer and design trend forecasters, content creators, designers, data analysts, advisory consultants and much more, united by a common goal: to create tomorrow.

WGSN's trusted consumer and design forecasts power outstanding product design, enabling our customers to create a better future. Our services cover consumer insights, beauty, consumer tech, fashion, interiors, lifestyle, food and drink forecasting, data analytics and expert advisory. If you are an expert in your field, we want to hear from you.

Role overview

WGSN is expanding its AI & Data capability and strengthening its data foundation. As a Data Engineer, you will play a key role in building, optimising, and maintaining the data pipelines, models, and infrastructure that power our classification systems, AI workflows, forecasting models, TikTok insights, and consumer intelligence products.

You will work closely with senior data scientists, analysts, and engineers, particularly within the TikTok and Pulse pods, ensuring high-quality, well-modelled, reliable data flows across Snowflake, Databricks, and downstream systems. This role is ideal for someone with strong technical depth, advanced SQL and data modelling capabilities, and a passion for building scalable, efficient data systems. This is a hands‑on, senior individual‑contributor role requiring at least 5+ years of experience in data engineering and the ability to mentor junior engineers when needed.

Key accountabilities

Data Architecture & Modelling

• Design, develop, and maintain scalable data architectures across Snowflake, Databricks, and cloud environments.
• Lead schema design, dimensional modelling, and query optimisation to support high-performance analytics and AI workloads.
• Collaborate with senior data scientists to structure data for classification, forecasting, embedding generation, and multimodal workflows.

Advanced SQL & Performance Optimisation

• Own complex SQL development and performance tuning across DS&E and DPS teams.
• Optimise costly queries, improve warehouse efficiency, and ensure best-practice SQL standards across shared codebases.

Pipeline Development (Batch & Near-Real-Time)

• Build robust ETL/ELT pipelines for ingestion, transformation, validation, and delivery.
• Develop resilient ingestion workflows for external APIs, including rate limiting, retries, schema drift handling, and monitoring.
• (Future-facing) Support design of streaming or near-real-time data flows as product needs evolve.

Snowflake & Databricks Expertise

• Implement pipelines using Snowpark, PySpark, and distributed compute environments.
• Apply Snowflake performance optimisation, cost governance, RBAC, and Snowflake best practices.
• Support compute scaling across cloud platforms (AWS, GCP) and distributed cluster environments.

Data Quality, Contracts & Observability

• Implement data validation frameworks (e.g., Great Expectations) and enforce data contracts.
• Build monitoring, alerting, and lineage visibility for pipelines (e.g., dbt tests, metadata tracking).
• Ensure high standards of data accuracy, completeness, and reliability.

DataOps & CI/CD for Data

• Build automated CI/CD workflows for data using GitHub Actions, CircleCI, or similar.
• Develop automated unit tests, integration tests, and quality gates for data pipelines.
• Partner with DataOps & Platform Engineering to improve observability, documentation, and deployment workflows.

Workflow Orchestration

• Build and maintain orchestration workflows using Airflow, Prefect, Dagster, or equivalent.
• Optimise DAGs for performance, reliability, and clarity, while ensuring operational excellence.

Cloud Infrastructure, Containers & Runtime Management

• Run, log, monitor, and debug workloads across VMs, Docker containers, and cloud compute environments.
• Improve reliability and maintainability of containerised workloads powering AI and data pipelines.

Cross-Functional Collaboration

• Translate analytical and AI requirements into scalable engineering solutions.
• Document pipelines, decisions, runbooks, and architecture clearly and consistently.

Mentorship & Capability Building

• Provide guidance to junior engineers and contribute to building team-wide engineering maturity.

This list is not exhaustive and there may be other activities you are required to deliver.

Skills, experience & qualifications required

Experience

• 5+ years of hands-on experience as a Data Engineer.
• Proven success designing and scaling production-grade data pipelines in cloud environments.
• Experience mentoring junior engineers or contributing to capability uplift across teams.

Technical Skills

• Expert-level SQL: complex queries, optimisation, performance tuning, analytical SQL.
• Advanced data modelling (star schemas, normalisation, dimensional modelling).
• Strong Python skills, including Pandas, NumPy, and PySpark/Snowpark.
• Experience with Snowflake (performance optimisation, cost management, RBAC, governance).
• Experience with Databricks, distributed compute, and PySpark.
• Data pipeline orchestration (Airflow, Dagster, Prefect).
• Data validation frameworks (e.g., Great Expectations).
• Strong familiarity with cloud platforms (AWS or GCP).
• Experience building resilient API ingestion pipelines.
• Understanding of Docker, Linux servers, and cloud VMs.
• DataOps & DevOpsCI/CD workflows for data pipelines (GitHub Actions, CircleCI).
• Logging, monitoring, observability for data workflows.

Soft Skills

• Excellent communication across technical and non-technical teams.
• Ability to work within and contribute to cross-functional pods (DS + DE + Product + Content).
• Strong problem-solving skills and ownership mindset.

What we offer

Our benefits and wellbeing package offers flexible benefits you can tailor to your own personal needs, including:

• 25 days of holiday per year – with an option to buy/sell up to 5 days
• Pension, Life Assurance and Income Protection – Flexible benefits platform with options including Private Medical, Dental Insurance & Critical Illness
• Employee assistance programme, season ticket loans and cycle to work scheme
• Volunteering opportunities and charitable giving options
• Great learning and development opportunities.

More about WGSN

WGSN is the global authority on consumer trend forecasting.

We help brands around the world create the right products at the right time for tomorrow’s consumer.

Our values

We Are Everywhere

The future is everything, it happens everywhere. WGSN is the world-leading forecaster because we track and analyse consumer behaviours, product innovation, design and creativity, everywhere.

We Are Future Focused

We utilise our global resources and intelligence to research, source and analyse quantitative and qualitative data to produce our forecasts. Everything we do is focused on working with our customers to create a successful and positive tomorrow.

We Are Rigorous

We source, review and assess quantitative and qualitative data to produce robust, actionable forecasts. To provide credible insights and design solutions for our clients, it is essential that rigour runs through everything we do.

Our culture

An inclusive culture is one of our key priorities. We want our people to truly be themselves and thrive. We love having a diverse team of people who bring new ideas, different strengths and perspectives & reflect the global audience we work with.

Inclusive workforce

We are committed to supporting the environment and sustainability, including ensuring our pension plan defaults to sustainable options and striving to be net zero by 2030.

Recognising great performance

Our awards schemes recognise and reward the brilliant achievements of our people.

We offer a flexible working environment with a wide range of flexible, hybrid and agile working arrangements. Conversations about flexible working have always been—and will continue to be—actively encouraged here, but we do not offer full remote working.

We want to ensure everyone has the opportunity to perform their best when interviewing, so if you require any reasonable adjustments that would make you more comfortable during the process, please let us know so that we can do our best to support you.

A Note for Applicants

We use AI to help our team screen applications and identify candidates whose skills and experience match the role. This technology removes personal information to promote a fair and unbiased process. We believe this tool helps us find the best talent while maintaining transparency and fairness.

A Note for Recruiters

Thank you so much for your interest in working with us at WGSN! Our internal Talent Acquisition team takes care of all our recruitment efforts. When we need some extra help, we partner with agencies on our Preferred Supplier List (PSL) that truly understand our business, culture and ways of working together. Since we focus on these established partnerships, we’re unable to respond to unsolicited contacts or CVs from outside our PSL. But don’t worry! If we decide to explore new partnerships, we’ll be sure to reach out.