Azure Databricks Data Engineer | ETL, Delta Lake & Governance

MPM Products Limited; Manchester; Hybrid; GBP 60,000 - 90,000; Full time

Job summary

MPM Products Limited in Manchester is seeking a Data Engineer to develop, maintain and optimise our data infrastructure, ensuring scalable, secure centralised data assets and reliable ETL pipelines using Azure Databricks.

You will lead end-to-end data workstreams, implement Bronze/Silver/Gold architecture, govern data access, document data dictionaries, coach colleagues and promote a data-first culture across the business.

Qualifications

• Databricks Data Engineer Associate certification or equivalent
• Experience building and maintaining data pipelines in Databricks
• Experience with Azure cloud environments
• Data Warehousing, Data Lakes, Delta Lake concepts
• Strong Python (PySpark, Pandas)
• Strong SQL for data retrieval/manipulation
• Strong problem-solving and communication skills
• Experience with GitHub and CI/CD pipelines (Azure DevOps, DABs)

Responsibilities

• Lead, design and maintain scalable ETL pipelines ingesting data from APIs and other sources
• Develop and implement Bronze/Silver/Gold medallion architecture on Databricks
• Monitor costs related to storage and compute for cost-effective resources
• Automate manual tasks with data-driven solutions to improve productivity
• Document data dictionaries and catalogues using Unity Catalog for LLM capabilities
• Establish strong data governance and access controls
• Coach and upskill colleagues to build data capability
• Coordinate end-to-end data workstreams with stakeholders
• Promote a data-first culture and Databricks as a single source of truth
• Contribute to best practices in the Databricks function

Skills

Python

SQL

Communication

Detail-oriented

Project management

Education

Databricks Data Engineer Associate

Tools

Databricks

Azure

GitHub

Azure DevOps

Delta Lake

Unity Catalog

CI/CD

Job description

MPM Products Limited in Manchester is seeking a Data Engineer to develop, maintain and optimise our data infrastructure, ensuring scalable, secure centralised data assets and reliable ETL pipelines using Azure Databricks.

You will lead end-to-end data workstreams, implement Bronze/Silver/Gold architecture, govern data access, document data dictionaries, coach colleagues and promote a data-first culture across the business.