Data Engineer

MPM Products Limited; Manchester; Hybrid; GBP 60,000 - 90,000; Full time

Job summary

MPM Products Limited in Manchester is seeking a Data Engineer to develop, maintain and optimise our data infrastructure, ensuring scalable, secure centralised data assets and reliable ETL pipelines using Azure Databricks.

You will lead end-to-end data workstreams, implement Bronze/Silver/Gold architecture, govern data access, document data dictionaries, coach colleagues and promote a data-first culture across the business.

Qualifications

• Databricks Data Engineer Associate certification or equivalent
• Experience building and maintaining data pipelines in Databricks
• Experience with Azure cloud environments
• Data Warehousing, Data Lakes, Delta Lake concepts
• Strong Python (PySpark, Pandas)
• Strong SQL for data retrieval/manipulation
• Strong problem-solving and communication skills
• Experience with GitHub and CI/CD pipelines (Azure DevOps, DABs)

Responsibilities

• Lead, design and maintain scalable ETL pipelines ingesting data from APIs and other sources
• Develop and implement Bronze/Silver/Gold medallion architecture on Databricks
• Monitor costs related to storage and compute for cost-effective resources
• Automate manual tasks with data-driven solutions to improve productivity
• Document data dictionaries and catalogues using Unity Catalog for LLM capabilities
• Establish strong data governance and access controls
• Coach and upskill colleagues to build data capability
• Coordinate end-to-end data workstreams with stakeholders
• Promote a data-first culture and Databricks as a single source of truth
• Contribute to best practices in the Databricks function

Skills

Python

SQL

Communication

Detail-oriented

Project management

Education

Databricks Data Engineer Associate

Tools

Databricks

Azure

GitHub

Azure DevOps

Delta Lake

Unity Catalog

CI/CD

Job description

Manchester/ PERMANENT/ SALARY: COMPETITIVE

Job Purpose

Responsible for developing, maintaining and optimising MPM’s data infrastructure. Provide technical expertise to ensure efficient data handling, ensuring availability, scalability and security of the centralised data asset. Ensure that all ETL pipeline and scheduled jobs run efficiently and effectively. To build a centralised data asset in which data is stored logically. Train colleagues in the use and capability of Azure Databricks.

Key Responsibilities

This role deals with a wide variety of tasks, including but not limited to the following:

• Lead, design and maintain scalable ETL pipelines that ingest, clean and aggregate raw data from various sources such as APIs, storing data appropriately
• Develop and implement medallion (Bronze/Silver/Gold) architecture across Azure Databricks
• Monitor and control costs related to data storage, compute, and other services to ensure efficient and cost-effective use of resources
• Automate manual tasks by leading data-driven solutions, transforming processes into efficient, system-driven operations that improve productivity and accuracy
• Create and update technical documentation, such as data dictionaries and catalogues, through Unity Catalog to support the data infrastructure through large language model (LLM) functionalities such as Genie
• Establish strong data governance, ensuring colleagues have the level of data access that they require
• Build capability across the data function by coaching and upskilling colleagues (pairing, knowledge-sharing, and clear documentation), helping reduce single points of failure and improving consistency.
• Coordinate delivery of assigned data workstreams end-to-end, working closely with stakeholders and the wider team to agree scope, manage actions and dependencies, and deliver high-quality outcomes.
• Be an advocate for a "data-first" approach across the business, championing the use of Databricks to create a single source of truth for data and metrics across the business
• Contribute to best practices and continuous improvement of the Databricks function

Knowledge, Skills and Experience

Background in Data Engineering recognised through certification (Databricks Data Engineer Associate or similar) or through demonstratable experience

Experience building and maintaining data pipelines in Databricks

Experience working with cloud environments, preferably Azure

Experience in Data Warehousing, Data Lakes, Blob Storage, Lakehouses and Data Modelling concepts (Kimball, star schemas)

Strong ability to write Python code, including experience with data manipulation libraries (PySpark, Pandas etc)

Strong ability to write SQL queries for data retrieval and manipulation

Strong problem-solving skills to address and resolve data-related issues.

Effective communicator at a technical & non-technical level.

IT literate use of Microsoft applications.

High attention to detail.

Ability to lead and manage projects, including scoping, planning, and executing tasks to meet deadlines.

Experience with source control and CI/CD (GitHub, Azure DevOps, DABs)

Data architecting, solutions architecting and experience optimising pipelines and Delta Lake

Databricks Data Engineer Professional certified