Senior Data Engineer | Azure Databricks | SC Cleared

IntelStack; Greater London; Hybrid; GBP 90,000 - 130,000; Full time

Job summary

IntelStack is hiring a Senior Data Engineer to build and scale an Azure Databricks platform powering national-economic analysis. You will own end-to-end pipelines from ingestion to production-ready datasets, working with economists and data scientists.

Based in London with a hybrid model (3 days on site), the role requires 8+ years in data engineering, strong PySpark/SQL, and active SC clearance. You will implement governance, Delta Lake, Unity Catalog, and CI/CD practices to ensure reliable

Qualifications

• 8+ years in data engineering, including 3+ years hands on with Azure Databricks.
• Strong Python and PySpark, backed by advanced SQL.
• Deep grounding in data warehousing, data modelling and integration patterns.
• Unity Catalog and Delta Lake in a production environment.
• Data quality and governance experience, ideally including Azure Purview.
• Git, Agile delivery and a collaborative approach to code review.
• Active SC clearance. This is a firm requirement and we are unable to sponsor or process new clearance for this role.

Responsibilities

• Design scalable ingestion pipelines from APIs, databases and financial data providers into Azure Databricks.
• Writing complex transformations in PySpark and Spark SQL, with Delta Lake as the storage foundation.
• Tuning Databricks workloads for performance and cost, and embedding quality checks and validation into every pipeline.
• Implementing governance in practice: Unity Catalog, Azure Purview, data lineage, metadata and monitoring.
• Automating deployments through CI/CD in Azure DevOps and promoting DevOps practice across the team.
• Working directly with economists and data scientists to turn analytical requirements into engineered solutions.

Skills

Python

PySpark

SQL

Data warehousing

Data modelling

Data governance

Agile delivery

Git

Tools

Azure Databricks

Delta Lake

Unity Catalog

Azure Purview

Azure DevOps

Job description

IntelStack is hiring a Senior Data Engineer to build and scale an Azure Databricks platform powering national-economic analysis. You will own end-to-end pipelines from ingestion to production-ready datasets, working with economists and data scientists.

Based in London with a hybrid model (3 days on site), the role requires 8+ years in data engineering, strong PySpark/SQL, and active SC clearance. You will implement governance, Delta Lake, Unity Catalog, and CI/CD practices to ensure reliable