Big Data Lead

Hexaware; United Kingdom; Hybrid; GBP 90,000 - 120,000; Full time

Job summary

Allianz is seeking a Senior Data/Platform Engineer to join our Data & Analytics team in the UK. The role blends hands-on data engineering on Databricks and Azure Synapse with platform administration across the cloud data estate.

You will design, build, and operate scalable data pipelines while maintaining the Azure platform, including IaC (Pulumi) and CI/CD automation, security, and governance. Strong Python/PySpark skills and Azure expertise are essential.

Qualifications

• Proficient in Python/PySpark data engineering
• Strong Azure platform knowledge and data tooling experience
• Experience mentoring teammates and leading solution design

Responsibilities

• Lead solution design activities and mentor junior colleagues
• Develop, test, and document scalable ETL/ELT pipelines using Databricks and Azure Synapse
• Administer Azure data platform components (Synapse, Databricks, ADLS Gen2, Key Vault)
• Manage infrastructure-as-code with Pulumi and related tools (Terraform/Bicep)
• Design and operate CI/CD pipelines using GitHub Actions and/or Azure DevOps
• Administer Databricks workspaces and governance features
• Monitor platform health via Azure Monitor and related telemetry

Skills

Python/PySpark

Cloud data engineering

Solution design

Mentoring

CI/CD practices

Azure platform knowledge

Security mindset

Tools

Databricks

Azure Synapse

Pulumi

Terraform

Bicep

GitHub Actions

Azure DevOps

Key Vault

ADLS Gen2

Job description

Overall Objectives of the Job We are seeking a skilled and experienced Senior Data / Platform Engineer to join our Data & Analytics team. This hybrid role combines hands‑on data engineering on Databricks and Azure Synapse with platform administration responsibilities across our cloud data estate. The role holder will design, build, and operate scalable data pipelines while also maintaining the underlying Azure platform — including infrastructure-as-code (Pulumi), CI/CD automation, monitoring, security, and Databricks workspace administration. The ideal candidate combines strong Python/PySpark engineering skills with deep Azure platform knowledge and a service‑excellence mindset, ensuring the Allianz Data Technology platform remains secure, reliable, and aligned to Allianz standards.

Duties and Responsibilities

• Lead solution design activities, collaborating with peers and mentoring junior colleagues to define and execute the team backlog.
• Develop, test, and document scalable ETL/ELT data pipelines and workflows using Databricks and Azure Synapse to ingest and transform data from a variety of sources.
• Administer and maintain Azure data platform components including Synapse, Databricks, ADLS Gen2, Key Vault, networking (VNets, NSGs, Managed Private Endpoints) and access control (RBAC, ACLs).
• Manage infrastructure-as-code across Dev, Staging, and Production environments using Pulumi (and equivalents such as Terraform / Bicep).
• Design and operate CI/CD pipelines using GitHub Actions (with OIDC federation) and/or Azure DevOps, supporting trunk‑based development practices.
• Administer Databricks workspaces — cluster policies, Secret Scopes, Repos/Git integration, Workflow job health, and Unity Catalog governance.
• Monitor platform and pipeline health using Azure Monitor, Log Analytics, KQL, and Azure Dashboards; triage and resolve incidents.