Senior Data Engineer - Databricks & Azure Lead (Hybrid)

vavemga; Manchester; Hybrid; GBP 70,000 - 110,000; Full time

Benefits offered by this job

Hybrid working

Competitive base salary

Pension

Health & dental cover

Job summary

Canopius is seeking a senior data engineer in Manchester to apply expertise in Azure technologies, data storage and data engineering to extend and maintain our data ecosystem. You will lead a small team, mentor junior engineers, and collaborate with stakeholders to deliver scalable data pipelines on Databricks using PySpark and Spark SQL.

The role involves guiding data strategy, migrating legacy processes, and delivering enterprise data solutions in a cloud-native environment, with hybrid

Qualifications

• Strong understanding of data engineering principles, data architecture, and cloud-based data platforms.
• Proficient in Python (PySpark) and SQL for building and optimising data transformations at scale.
• Strong hands-on Databricks experience.
• Experience with Azure data storage technologies (Azure SQL, Data Lakes, Blob Containers).
• Experience with data transformation and modelling techniques.
• Thorough understanding and experience with data warehousing concepts and dimensional modelling.
• Experience with ETL and orchestration tooling (Data Factory, Databricks).
• Experience with data governance and cataloguing using Databricks Unity Catalog (access control, lineage, discoverability).
• Experience with Azure DevOps, including branching and release/deployment strategies; Git, CI/CD and DevOps Pipelines.

Responsibilities

• Provide technical leadership and direction across data engineering initiatives.
• Design solutions for production operation, including documentation, monitoring, alerting, logging, access and defined support responsibilities.
• Design, build, and optimise data pipelines on Databricks using PySpark and Spark SQL, applying Delta Lake and a medallion/Lakehouse architecture.
• Implement data validation and reconciliation checks, and automated testing across pipelines to ensure trusted, accurate outputs.
• Support the migration and re-platforming of legacy SQL Server data warehouse processes as required.
• Challenge requirements constructively and provide guidance on what is technically feasible, scalable, and commercially viable.
• Communicate technical solutions, trade-offs, risks, and opportunities to both technical and non-technical audiences, managing expectations and influencing decisions with evidence-based recommendations.
• Lead technical planning activities, ensuring work is appropriately defined, estimated, and prioritised. Provide delivery estimates and communicate progress and blockers to stakeholders.
• Collaborate with Product Owners, Business Analysts, and Solution Architects to define product vision and technical roadmaps.
• Drive adoption of modern engineering practices, AI technologies, and automation opportunities across the team.
• Drive delivery of complex, end-to-end work items, seeking clarification of ambiguous specifications and proactively flagging risk of technical debt.
• Management of own task list and ensuring that plans are agreed.
• Line management of junior and mid-level data engineers in the team.
• Mentor and support engineers within the squad through pairing, peer review of pull requests, constructive feedback and knowledge sharing, ensuring successful delivery of sprint objectives and acting as a role model to the team.
• Undertake other ad-hoc duties as required.

Skills

Data engineering

Python (PySpark)

SQL

Databricks

Azure data storage

Azure Data Lakes

Tools

Databricks Unity Catalog

Azure Data Factory

Job description

Canopius is seeking a senior data engineer in Manchester to apply expertise in Azure technologies, data storage and data engineering to extend and maintain our data ecosystem. You will lead a small team, mentor junior engineers, and collaborate with stakeholders to deliver scalable data pipelines on Databricks using PySpark and Spark SQL.

The role involves guiding data strategy, migrating legacy processes, and delivering enterprise data solutions in a cloud-native environment, with hybrid