Senior Data Platform Engineer – Metadata-Driven Automation (Hybrid)

Perch Group; Lancashire; Hybrid; GBP 65,000 - 80,000; Full time

Benefits offered by this job

Hybrid work

Blackpool office

20% bonus

Formal qualifications sponsorship

Wellbeing benefits

Job summary

Perch Group in the UK is seeking a Senior Data Engineer to industrialise our data platform with metadata‑driven frameworks and repeatable patterns. You will design scalable ingestion and publishing pipelines using ADF, Databricks, and T-SQL, while raising engineering standards across the team.

You'll mentor engineers, advance CI/CD and data governance, and partner with product teams to deliver enterprise‑scale data solutions in a hybrid, Blackpool‑based role with opportunities to gain formal

Qualifications

• 5+ years advanced T-SQL with strong optimisation and engineering‑oriented design patterns.
• 5+ years ADF: parameterisation, custom frameworks, and template‑based orchestration.
• 5+ years Databricks including PySpark optimisation, Delta Lake architecture.
• Experience building metadata‑driven or config‑driven frameworks, not just talking about them.
• 5+ years data modelling, including dimensional models for enterprise‑scale consumption.
• End‑to‑end delivery experience across data warehouse or data platform programmes.
• Experience with dynamic schema detection, inferred metadata processing, or multi‑tenant ingestion patterns.
• Advanced orchestration approaches (Databricks Workflows, Functions, event‑driven patterns).
• Practical CI/CD experience using Azure DevOps for data engineering.
• Experience in regulated industries with quality, lineage, and governance requirements.
• Understanding of data management frameworks (DAMA, DCAM, etc.).

Responsibilities

• Design and implement metadata‑driven and config‑driven frameworks to replace one‑off pipelines with reusable, scalable patterns.
• Develop templates, orchestration patterns, and reusable components that minimise manual engineering effort.
• Introduce engineering standards prioritising automation, maintainability, and lifecycle governance.
• Build and optimise robust ingestion, transformation, and publishing pipelines using ADF, Databricks (PySpark/Delta Lake) and T-SQL—embed automation from the start.
• Assist in maturing the enterprise data warehouse architecture and data models.
• Ensure designs align with best practices around performance, logging, monitoring, and quality.
• Partner with architects and product teams to identify high‑value automation opportunities.
• Mentor engineers and uplift engineering discipline through coaching, patterns, and code reviews.
• Strengthen DevOps, CI/CD, and automated testing approaches for data workloads.

Skills

T-SQL

ADF

Databricks

Metadata-driven

Data modelling

CI/CD

Azure DevOps

Data governance

Tools

PySpark

Delta Lake

SQL Server

Job description

Perch Group in the UK is seeking a Senior Data Engineer to industrialise our data platform with metadata‑driven frameworks and repeatable patterns. You will design scalable ingestion and publishing pipelines using ADF, Databricks, and T-SQL, while raising engineering standards across the team.

You'll mentor engineers, advance CI/CD and data governance, and partner with product teams to deliver enterprise‑scale data solutions in a hybrid, Blackpool‑based role with opportunities to gain formal