Senior Data / Platform Engineer

N Consulting Global; Guildford; Hybrid; GBP 70,000 - 110,000; Full time

Job summary

N Consulting Global in Guildford is seeking a Senior Data/Platform Engineer to join our Data & Analytics team in a hybrid role. You will design, build, and operate scalable data pipelines on Databricks and Azure Synapse while also maintaining the Azure platform, including Pulumi, CI/CD, and security.

The role blends hands-on data engineering with platform administration. The ideal candidate has strong Python/PySpark skills and extensive Azure experience, with 6–10 years in data engineering.

Qualifications

• Bachelor's or Master's degree in Computer Science, Information Systems, or related field.
• 6–10 years of experience in data engineering and Azure platform administration.

Responsibilities

• Lead solution design activities with peers and mentor junior colleagues.
• Develop, test, and document scalable ETL/ELT pipelines using Databricks and Azure Synapse.
• Administer Azure data platform components (Synapse, Databricks, ADLS Gen2, Key Vault).
• Manage infrastructure-as-code with Pulumi across Dev, Staging, and Production.
• Design and operate CI/CD pipelines using GitHub Actions and/or Azure DevOps.
• Administer Databricks workspaces and governance components.
• Monitor platform health using Azure Monitor and Log Analytics; triage incidents.
• Implement robust data security and ensure data privacy compliance.
• Carry out routine platform operations: patching, backups, DR readiness, runbooks.

Skills

Databricks experience

Python/PySpark

CI/CD automation

Infrastructure-as-code

Scripting (PowerShell/Bash)

Data security

Education

Bachelor's or Master's in CS/IS

Tools

Databricks

Azure Synapse

ADLS Gen2

Key Vault

GitHub Actions

Azure DevOps

Pulumi

Terraform

Bicep

Job description

Location: Guildford Business Park Guildford Surrey GU2 8XG

Job Type : Open with FTE/Contract (Initially with 6 months)

Work Mode : Remote with occasional office visit

Job Description:

Overall Objectives of the Job

We are seeking a skilled and experienced Senior Data / Platform Engineer to join our Data & Analytics team. This hybrid role combines hands‑on data engineering on Databricks and Azure Synapse with platform administration responsibilities across our cloud data estate. The role holder will design, build, and operate scalable data pipelines while also maintaining the underlying Azure platform — including infrastructure‑as‑code (Pulumi), CI/CD automation, monitoring, security, and Databricks workspace administration. The ideal candidate combines strong Python/PySpark engineering skills with deep Azure platform knowledge and a service‑excellence mindset.

Duties and Responsibilities

Lead solution design activities, collaborating with peers and mentoring junior colleagues to define and execute the team backlog.

Develop, test, and document scalable ETL/ELT data pipelines and workflows using Databricks and Azure Synapse to ingest and transform data from a variety of sources.

Administer and maintain Azure data platform components including Synapse, Databricks, ADLS Gen2, Key Vault, networking (VNets, NSGs, Managed Private Endpoints) and access control (RBAC, ACLs).

Manage infrastructure‑as‑code across Dev, Staging, and Production environments using Pulumi (and equivalents such as Terraform / Bicep).

Design and operate CI/CD pipelines using GitHub Actions (with OIDC federation) and/or Azure DevOps, supporting trunk‑based development practices.

Administer Databricks workspaces — cluster policies, Secret Scopes, Repos/Git integration, Workflow job health, and Unity Catalog governance.

Monitor platform and pipeline health using Azure Monitor, Log Analytics, KQL, and Azure Dashboards; triage and resolve incidents.

Implement robust data security and ensure compliance with data privacy regulations; manage service principals, Managed Identities, and least‑privilege access.

Carry out routine platform operations: patching, backups, storage lifecycle, tagging, access reviews, DR readiness, and runbook execution.

Identify and address performance bottlenecks and data quality issues to ensure data accuracy and reliability.

Work with testers to ensure automated test plans are in place and agree test packs for UAT; review peers' work and take accountability for the quality of squad deliverables.

Collaborate with stakeholders and analysts to understand data requirements and deliver clean, reliable, accessible data.

Qualification, Experience, Technical and Functional Skills

Bachelor's or Master's degree in Computer Science, Information Systems, or a related field, with 6–10 years of relevant experience in data engineering and Azure platform administration.

Must Have

Databricks: hands‑on experience building and optimizing pipelines, managing Delta Lake, and administering workspaces (cluster policies, Unity Catalog, Secret Scopes, Workflows).

Python / PySpark: strong programming skills for data processing, automation, and scripting.

Azure data stack: Synapse, Databricks, ADLS Gen2, Key Vault — including Linked Services, Managed Identity, and Spark Pool configuration.

Azure platform fundamentals: compute, storage, networking (VNets, NSGs, Private Endpoints), identity and RBAC.

CI/CD: GitHub Actions (with OIDC federation) and/or Azure DevOps for data and platform deployments.

Infrastructure‑as‑code: Pulumi (or Terraform / Bicep) across multiple environments.

Scripting: PowerShell and Bash for platform automation.

Big data file formats: Parquet and Delta Lake.

Cloud‑native data modelling and ETL/ELT frameworks on Azure.

Good to Have

Data governance: lineage, cataloguing, sensitivity labels.

IDMC Secure Agent, Power Automate flows.

Security baselines: CIS / NIST.

Observability tooling: OpenTelemetry, Datadog.

AI tools and their application in data engineering.