Data Platform Lead

Solution Performance Resourcing Limited; United Kingdom; On-site; GBP 95,000 - 130,000; Full time

Job summary

SPG Resourcing is seeking a Senior Data Platform Lead to own and improve a live data environment in the UK. You will drive incident diagnosis, recovery, and platform resilience while leading hands-on work across PySpark, Delta Lake, and Databricks Workflows.

In this high-impact role you will collaborate with ERP data sources, manage production changes, and evolve runbooks for operational readiness in a 17x5 support model.

Qualifications

• Lead incidents and diagnose data-flow interruptions across ERP to downstream applications.
• Own platform health, monitoring, performance, cost optimisation and resilience.
• Hands-on leadership with PySpark, Spark SQL, Delta Lake and Databricks Workflows.

Responsibilities

• Lead technical diagnosis and recovery when incidents affect data flowing from ERP systems through ingestion, Azure storage and Databricks to downstream applications.
• Own platform health, monitoring, performance, cost optimisation and operational resilience.
• Provide hands-on leadership across PySpark, Spark SQL, Delta Lake and Databricks Workflows.
• Set standards for Unity Catalog, access controls, data lineage and production releases.
• Work with engineering, application, service management and vendor teams to resolve complex issues and prevent recurrence.
• Improve runbooks, recovery procedures and architecture documentation.
• Transfer knowledge to the wider team and support operational readiness.
• Participate in a 17x5 support model, with the working pattern confirmed during the briefing process.

Skills

PySpark

Spark SQL

Delta Lake

Databricks Workflows

Azure Databricks

ADLS Gen2

Azure DevOps

Git

CI/CD

Data flow troubleshooting

ERP data integration

JD Edwards

Oracle ERP

SAP

Dynamics

Incident leadership

Qlik Replicate

CDC

Supply chain data

Tools

Azure DevOps

Git

CI/CD pipelines

Job description

We are recruiting for a Senior Data Platform Lead opportunity within a live, production-focused data environment, ideal for a technical leader who can protect service stability while improving platform performance and resilience.

This company is seeking someone who can take ownership when data stops flowing, drive diagnosis and recovery across complex pipelines, and keep critical business applications supported. The Senior Data Platform Lead will play a hands-on role across incident response, operational health, and continuous improvement, making this a high-impact opportunity for a candidate who thrives under pressure.

Responsibilities

• Lead technical diagnosis and recovery when incidents affect data flowing from ERP systems through ingestion, Azure storage and Databricks to downstream applications.
• Own platform health, monitoring, performance, cost optimisation and operational resilience.
• Provide hands-on leadership across PySpark, Spark SQL, Delta Lake and Databricks Workflows.
• Set standards for Unity Catalog, access controls, data lineage and production releases.
• Work with engineering, application, service management and vendor teams to resolve complex issues and prevent recurrence.
• Improve runbooks, recovery procedures and architecture documentation.
• Transfer knowledge to the wider team and support operational readiness.
• Participate in a 17x5 support model, with the working pattern confirmed during the briefing process.

Experience & Skills

• Strong, recent experience supporting Azure Databricks in production, including ownership of incidents and service restoration.
• Hands-on capability with PySpark, SQL, Delta Lake, Workflows and Unity Catalog.
• Experience with ADLS Gen2, Azure DevOps, Git, CI/CD and controlled production changes.
• Ability to troubleshoot data flows end to end, from transactional source systems through ingestion and transformation to the final consumer.
• Experience integrating ERP data, such as JD Edwards, Oracle ERP, SAP or Dynamics.
• Confidence to lead a major incident, explain business impact clearly and make sound recovery decisions under pressure.
• Useful experience in Qlik Replicate, CDC, supply chain data or manufacturing environments.
• Ability to provide a brief example of a production Databricks incident personally diagnosed and resolved.

Additional

• Right to work in the UK required
• Equal opportunities employer

Equal Opportunity Statement:

SPG Resourcing is an equal opportunities employer and is committed to fostering an inclusive workplace which values and benefits from the diversity of the workforce we hire. We offer reasonable accommodation at every stage of the application and interview process.