Senior Data Engineer

IntelStack; Greater London; Hybrid; GBP 90,000 - 130,000; Full time

Job summary

IntelStack is hiring a Senior Data Engineer to build and scale an Azure Databricks platform powering national-economic analysis. You will own end-to-end pipelines from ingestion to production-ready datasets, working with economists and data scientists.

Based in London with a hybrid model (3 days on site), the role requires 8+ years in data engineering, strong PySpark/SQL, and active SC clearance. You will implement governance, Delta Lake, Unity Catalog, and CI/CD practices to ensure reliable

Qualifications

• 8+ years in data engineering, including 3+ years hands on with Azure Databricks.
• Strong Python and PySpark, backed by advanced SQL.
• Deep grounding in data warehousing, data modelling and integration patterns.
• Unity Catalog and Delta Lake in a production environment.
• Data quality and governance experience, ideally including Azure Purview.
• Git, Agile delivery and a collaborative approach to code review.
• Active SC clearance. This is a firm requirement and we are unable to sponsor or process new clearance for this role.

Responsibilities

• Design scalable ingestion pipelines from APIs, databases and financial data providers into Azure Databricks.
• Writing complex transformations in PySpark and Spark SQL, with Delta Lake as the storage foundation.
• Tuning Databricks workloads for performance and cost, and embedding quality checks and validation into every pipeline.
• Implementing governance in practice: Unity Catalog, Azure Purview, data lineage, metadata and monitoring.
• Automating deployments through CI/CD in Azure DevOps and promoting DevOps practice across the team.
• Working directly with economists and data scientists to turn analytical requirements into engineered solutions.

Skills

Python

PySpark

SQL

Data warehousing

Data modelling

Data governance

Agile delivery

Git

Tools

Azure Databricks

Delta Lake

Unity Catalog

Azure Purview

Azure DevOps

Job description

Senior Data Engineer | Azure Databricks | SC Cleared | London (Hybrid, 3 days on site)

Build the data platform behind national economic analysis.

We are hiring a Senior Data Engineer to develop and scale an Azure Databricks platform that powers monetary analysis, economic forecasting and modelling. The data you engineer feeds the models that inform decisions at a national level, so quality, lineage and reliability genuinely matter here.

This is hands on engineering, not oversight. You will own pipelines end to end, from ingestion through transformation to governed, production ready datasets used daily by economists and data scientists.

What you will be doing

• Designing and building scalable ingestion pipelines from APIs, relational databases and financial data providers into Azure Databricks
• Writing complex transformations in PySpark and Spark SQL, with Delta Lake as the storage foundation
• Tuning Databricks workloads for performance and cost, and embedding quality checks and validation into every pipeline
• Implementing governance in practice: Unity Catalog, Azure Purview, data lineage, metadata and monitoring
• Automating deployments through CI/CD in Azure DevOps and promoting DevOps practice across the team
• Working directly with economists and data scientists to turn analytical requirements into engineered solutions

What we need from you

• 8+ years in data engineering, including 3+ years hands on with Azure Databricks
• Strong Python and PySpark, backed by advanced SQL
• Deep grounding in data warehousing, data modelling and integration patterns
• Unity Catalog and Delta Lake in a production environment
• Data quality and governance experience, ideally including Azure Purview
• Git, Agile delivery and a collaborative approach to code review
• Active SC clearance. This is a firm requirement and we are unable to sponsor or process new clearance for this role

Nice to have

• Streaming experience with Kafka or Azure Event Hubs
• Azure Data Engineer Associate or Databricks certification
• Docker, Kubernetes, Terraform or infrastructure as code

Why this role

You get a modern, well funded Azure estate, a mandate to build it properly rather than patch it, and work with real public significance. Hybrid working, three days on site in central London.