Senior Data Engineer

AB Agri Ltd; Peterborough; Remote; GBP 50,000 - 80,000; Full time

Job summary

An established industry player is seeking a Senior Data Engineer to enhance their cloud-based data architecture. In this dynamic role, you will design and optimize data pipelines using cutting-edge technologies like Azure and Databricks. You'll collaborate with cross-functional teams to ensure data quality and compliance while driving innovation through automation. This remote position offers a modern work environment with occasional visits to the office, where you'll be part of a culture that values inclusivity and continuous learning. If you're passionate about data engineering and looking to make a significant impact, this opportunity is for you.

Benefits

Free onsite parking

Complimentary drinks

Daily free lunch option

Ongoing career development

Annual salary reviews

Incentive bonus

Qualifications

• Experience in cloud-native data solutions with Azure and Databricks.
• Proficient in SQL and Python for developing data pipelines.

Responsibilities

• Design and maintain scalable data pipelines for data-driven initiatives.
• Collaborate with analysts and engineers to deliver data solutions.

Skills

Data Engineering

Azure

Databricks

Spark

SQL

Python

Data Architecture

Lakehouse Architecture

Delta Lakes

Problem-Solving

Education

Bachelor's Degree in Computer Science or related field

Certifications in Azure Data Engineering

Tools

Microsoft Fabric

Azure Data Factory

Synapse Analytics

Logic Apps

Event Hub

Job description

The Role & Key Responsibilities

Job Title:Senior Data Engineer

Location: Remote

Contract and Shift details:Monday to Friday, 37.5 hours a week

Are you a motivated Data Engineer ready to enhance data architecture in a dynamic, cloud-first environment? Do you thrive on building modern, scalable data platforms and forward-thinking analytics solutions?

Our Data & Analytics team is looking for a skilled Data Engineer to help design, develop, and optimise our cloud-based data infrastructure. You’ll play a key role in unlocking data-driven insights across the ABF group, working with the latest technologies like Microsoft Fabric, Databricks, and Spark to support real-time and batch data needs across global operations.

What you’ll be doing:

• Designing, building and maintaining robust, scalable data pipelines using Azure, Databricks, and Microsoft Fabric to support current and future data-driven initiatives across the business.
• Developing and optimising our data architecture, ensuring it is efficient, secure, and aligned with evolving analytics and reporting requirements.
• Partnering closely with data analysts, scientists, and software engineers to understand their data needs and deliver tailored solutions that enable actionable insights.
• Supporting both batch and real-time data processing scenarios by leveraging Spark, Delta Lakes, and Lakehouse architecture to deliver consistent and reliable data flows.
• Building reusable frameworks and internal tools that streamline data ingestion, transformation, and analysis across a variety of structured and unstructured data sources.
• Maintaining data quality, security, and compliance by embedding governance practices into the data engineering lifecycle, particularly in cloud and multi-region environments.
• Diagnosing and resolving complex data issues across platforms, performing root cause analysis, and implementing long-term improvements to ensure system reliability and scalability.
• Identifying and implementing opportunities for automation and infrastructure optimization, contributing to a culture of continuous improvement within the Data & Analytics team.

What you’ll bring:

• Experience in designing and developing cloud-native data solutions, with solid understanding of contemporary data platforms including Azure, Databricks, and Microsoft Fabric.
• Deep practical experience with Spark, along with solid knowledge in batch and real-time data processing frameworks.
• Proficient in SQL and Python programming, with experience developing performant and reusable data pipelines across both structured and unstructured data sources.
• A solid understanding of Lakehouse architecture, Delta Lakes, and Data Lake Storage (Gen2), including designing data models and storage strategies for scalability and performance.
• Familiarity with Azure tools such as Data Factory, Synapse Analytics, Logic Apps, and Event Hub, and experience integrating these into enterprise-grade data ecosystems.
• Strong problem-solving skills, with the ability to troubleshoot complex data issues, perform root cause analysis, and recommend practical technical improvements.
• A collaborative mindset with excellent communication skills — able to translate technical concepts for non-technical stakeholders and work effectively in cross-functional teams.
• A commitment to continuous learning and innovation, ideally demonstrated through certifications (Fabric Associate, Azure Data Engineer) or personal projects in cloud data engineering.

Location

This role is predominantly home-based and can be performed remotely - occasional visits to our recently refurbished Peterborough office on Lynchwood Business Park may be required. The office offers a modern working environment, excellent public transport links, free onsite parking, complimentary drinks, a daily free lunch option, and a welcoming team culture.

#LI-Remote

About The Company

Rewarding your passion

When you join us, you'll do meaningful work and be rewarded fairly for it.From annual salary reviews and incentive bonus, ongoing career development and enhanced family, financial and wellbeing benefits – these are just some of the ways we reward you - from day one.

About us

We’re AB Agri, part of Associated British Foods (ABF plc) and we’re passionate about feeding the world responsibly and affordably by making quality animal feed, creating nutrition and technology-based products, and offering data services for the agri-food industry. What sets us apart is how we delight our customers - and that’s because of great people like you. People with different life experiences who share common values. Together we’re building a fair, innovative and inclusive culture.

As a Disability Confident employer, we’re committed to ensuring our application and recruitment processes work for everyone so if you need any additional support (like alternative formats for your application, bringing a support dog to the offices, a sign language interpreter - or anything else) just let us know.

Application Notes

We hope you feel that AB Agri is somewhere you can thrive and so we’d love to hear from you. We get a lot of applications for our roles and sometimes need to close the process early so register your details and upload your CV via our careers website as soon as you can – it will only take a few minutes.

When you do, we’ll ask you some equal opportunities questions just to ensure our recruitment process is inclusive – or show us areas we can improve. This information is anonymised, separate from your application and is never seen by the hiring panel. You can choose not to answer these questions and this won’t affect your application.

Apply today or contact our recruitment team for more information.

Agencies and media sales

AB Agri group politely requests no contact from recruitment agencies or media sales. We don’t accept speculative CVs from recruitment agencies nor accept the fees associated with them.