Data Engineer

Compleat Food Group; United Kingdom; Hybrid; GBP 45,000; Full time

Job summary

A leading food company is seeking an experienced Data Engineer to join their team, focusing on data integration, transformation, and quality assurance for their Trade Promotion Management system. With a strong emphasis on modern data platforms and cloud technologies, you will play a crucial role in optimizing data workflows and supporting strategic business decisions. This hybrid role may require travel to national sites.

Qualifications

• 5+ years' experience in a Data Engineering role, preferably in FMCG/manufacturing.
• Strong expertise in modern data platforms and cloud technologies.
• Proven experience with real-time data streaming and data quality tools.

Responsibilities

• Build data pipelines to ingest data from multiple sources.
• Integrate with TPM platforms and ensure data integrity.
• Automate pipelines and collaborate with analysts.

Skills

Data Integration

Data Transformation

Data Modelling

Cloud Technologies (Azure preferred)

SQL

Python Programming

Real-time Data Streaming

Data Quality Assurance

Tools

Data lakehouse platforms (Databricks, Snowflake)

ETL/ELT tools (Azure Data Factory, Airflow)

Analytics platforms (Power BI)

Job description

Job Title: Data Engineer
Location: Hybrid/Remote but with travel to national sites
Salary: £45-50k
Contract: 6-month FTC

About the Role

We are looking for an experienced Data Engineer to play a pivotal role in the deployment of our Trade Promotion Management (TPM) system - UpClear BluePlanner. This is a highly data-intensive programme where your expertise will ensure the right data flows seamlessly into and out of the platform, supporting critical business activities such as forecasting, ROI calculations, accruals, and performance measurement.

If you thrive on building robust data pipelines, integrating complex systems, and enabling data-driven decision-making at scale, this role is for you.

Key Responsibilities

Data Integration & Connectivity

• Build pipelines to ingest data from multiple sources including ERP (SAP, Oracle), CRM, POS/syndicated data providers (Nielsen, IRI, Retail Link), and financial systems.
• Integrate with TPM platforms such as Salesforce TPM, SAP TPM, o9, Blacksmith, Eversight, and Anaplan.

Data Transformation & Mapping

• Harmonise product and customer hierarchies across systems.
• Apply business logic for promotion performance such as lift factors, cannibalisation, baseline vs incremental volume, and accrual/ROI calculations.

Data Modelling & Storage

• Design and optimise data models for promotion master data, actuals vs forecasts, claim reconciliation, and ROI KPIs.
• Ensure data warehouse/lake performance supports reporting dashboards and AI/ML models.

Data Quality & Validation

• Develop validation rules to ensure data integrity (e.g., promotion timelines, duplicates, account-product checks).
• Create monitoring and logging frameworks for errors and mismatches.

Ongoing Maintenance & Enhancements

• Automate pipelines using tools such as Airflow, Azure Data Factory, or Informatica.
• Adapt models and flows as business requirements evolve.
• Collaborate with TPM analysts and business users to refine outputs and troubleshoot issues.

Skills, Experience & Knowledge

Essential

• 5+ years' experience in a Data Engineering role (FMCG/manufacturing background highly desirable).
• Strong expertise in modern data platforms and cloud technologies (Azure preferred; AWS/GCP also considered).
• Proven hands-on experience with:
  • Data lakehouse platforms (Databricks, Snowflake, etc.)
  • ETL/ELT tools (dbt, Azure Data Factory, Airflow, Informatica, Talend)
  • SQL and programming (Python, Spark)
  • Real-time data streaming (Kafka, Azure Event Hub)
  • Data quality tools (Great Expectations, dbt tests, custom frameworks)
• Strong understanding of ERP integration (Infor ION, REST/SOAP APIs, flat file interfaces).
• Experience with analytics platforms such as Power BI.
• Knowledge of DevOps, CI/CD, and infrastructure-as-code principles for data pipelines.

Why Join Us?

This is an exciting opportunity to make a real impact in a fast-paced, data-driven environment. You'll be at the forefront of shaping how we integrate, manage, and leverage data to drive smarter trade promotion strategies and better business outcomes.