Data Engineering Manager

PEI Group; Greater London; Hybrid; GBP 90,000 - 120,000; Full time

Job summary

PEI Group is seeking a Data Engineering Manager in London to lead a multi-disciplinary team including data engineers, QA engineers, data analysts and Dynamics 365 developers. The role is 80% managerial with 20% hands-on delivery, focusing on data quality, governance, and scalable architecture.

You will drive standards, mentor engineers, and collaborate with product and operations to deliver data-driven solutions across the organization.

Qualifications

• 8+ years in data engineering or software development.
• Minimum 3 years people management.
• Experience hiring, mentoring and performance management.
• Proven track record leading multi-disciplinary teams.
• Strong understanding of data and software delivery lifecycle.

Responsibilities

• Plan, prioritise and deliver data initiatives with stakeholders.
• Manage delivery pipeline across multiple workstreams.
• Enforce coding standards, PR reviews, testing and branching.
• Drive data quality, governance, and platform reliability.
• Lead Databricks and Dynamics 365 workstreams.
• Hire, mentor and performance-manage engineers.
• Collaborate with data science, product, finance, operations.
• Contribute to architecture reviews and debugging production issues.
• Drive continuous improvement through tooling and automation.
• Research new technologies and trends in data engineering.
• Prioritise ad-hoc requests alongside projects.

Skills

Delivery management

Leadership

Agile

Communication

Tools

Databricks

Azure Data Factory

Dataverse

Delta Lake

PySpark

Databricks Workflows

Job description

About The Role

As a Data Engineering Manager, you will provide technical and delivery leadership to a multi-disciplinary team of data engineers, QA engineers, data analysts, and Dynamics 365 developers. You will drive coding standards, delivery excellence, and continuous improvement across the data platform. This is a leadership-focused role with an 80% managerial and 20% hands-on split.

Roles and Responsibilities

• Collaborate with Product, Project, and Business stakeholders to plan, prioritise, and deliver data initiatives on time and to a high standard, with adherence to standard processes.
• Manage the delivery pipeline across multiple workstreams, balancing capacity, dependencies, and competing priorities while maintaining a high bar for quality.
• Own and enforce coding standards, pull request review policies, testing practices, and branching strategies across the team.
• Drive data quality, governance, and platform reliability — including data cataloguing, lineage tracking, PII handling, and data quality SLA ownership.
• Optimise data solutions for maintainability, quality, performance, security, and scalability.
• Provide technical leadership and direction across Databricks and Dynamics 365 workstreams, setting clear expectations and objectives for individuals and the wider team.
• Hire, mentor, and performance‑manage engineers and analysts at all career levels, fostering growth, accountability, and a culture of continuous learning.
• Collaborate cross‑functionally with data science, product, finance, and business operations teams to deliver data‑driven solutions.
• Contribute hands‑on through architecture reviews, code reviews, prototyping, and debugging complex production issues — leading by example and staying close to the technology without being on the critical path.
• Drive a continuous improvement process — introduce tooling, automation, and process changes grounded in real data analysis and team feedback.
• Research and evaluate new technologies and trends in data engineering, recommending pragmatic improvements and innovations.
• Prioritise and manage ad‑hoc requests in parallel with ongoing projects.

About You

Requirements:

Qualification & Experience

• 8+ years of hands‑on experience in data engineering or software development, with significant exposure to data platforms and pipelines.
• A minimum of 3 years of people management experience; has built and operated teams of highly skilled engineers and analysts.
• Experience with hiring, mentoring, upskilling, and performance management.
• Proven track record of successfully managing and leading multi-disciplinary engineering teams.
• Solid understanding of the software and data delivery lifecycle.

Technical Skills

• Strong experience with Databricks (PySpark/Spark SQL, Delta Lake, Unity Catalog, medallion architecture, Databricks Workflows).
• Working knowledge of Microsoft Dynamics 365 data structures, Dataverse, and common integration patterns (Synapse Link for Dataverse, Dataverse APIs).
• Solid understanding of the Azure data ecosystem (Azure Data Factory, ADLS Gen2, Azure DevOps, Key Vault).
• Strong grasp of testing methodologies, CI/CD pipelines, and data quality automation.
• Experience embedding data governance practices — cataloguing, lineage, RBAC, and regulatory compliance (e.g. GDPR).

Personal Attributes

• Exceptional delivery management skills, including planning, executing, and delivering complex data projects across multiple workstreams.
• Ability to operate in an Agile environment with a start‑up mentality and comfort in an evolving, unstructured environment.
• Must be a compelling and clear communicator, able to represent the team to internal and external audiences with differing levels of technical fluency.
• Driving a high‑performance, collaborative, and inclusive team culture in a fast‑paced environment.

About Us

Who we are:

PEI Group is a subscriber‑focused business intelligence company. With our multi‑talented global team of over 490 people, spread across EMEA, USA & Asia, our purpose is to inform and connect investment professionals across global, specialised markets. We identify specific high growth, high value investment sectors and themes where deep insight, strong market relationships and active capital flows are critical for success.

What we do:

PEI Group provides industry‑leading journalism, data, and market insight to subscribing clients via a wide portfolio of specialist brands supported by our robust and scalable digital publishing, analytics, and database platform. We also track the firms and individuals who shape markets and bring client‑communities together to enable knowledge sharing, profile building and relationship development through professional networks and events. Wherever our markets are active – in New York, Los Angeles, Tokyo, Sydney, Hong Kong, Singapore, London and elsewhere – PEI is hard at work examining crucial market forces and shifting investment themes, identifying active investors and their capital allocations, and scanning ahead for regulatory changes, new compliance requirements and other risk factors.

At PEI we value diverse talent and welcome applications from everyone – regardless of background. We are an equal opportunity employer and our inclusive culture at PEI is reflected in every stage of the recruitment journey. Please inform us at initial stages of the recruitment process if you require any reasonable adjustments and we can accommodate this.

PEI Group supports flexible working arrangements, and we welcome career returners.