Python Data Engineer - Azure

Montash; Remote; GBP 60,000 - 80,000; Full time

Job summary

A leading consulting agency is looking for an SC Cleared Python Data Engineer to design and deliver scalable data pipelines using PySpark and Azure services. The ideal candidate will have strong Python expertise and experience in a cloud-native environment. Responsibilities include developing data ingestion pipelines, optimising Delta Lake transactions, and collaborating with multi-disciplinary teams. This remote position offers a challenging and dynamic work environment.

Qualifications

• Strong Python skills for production-quality code.
• Experience with PySpark and performance tuning.
• Familiarity with Behave for testing frameworks.
• Understanding of Delta Lake and its transactions.
• Experience with Docker in development workflows.
• Hands-on experience with Azure services.

Responsibilities

• Design, develop, and maintain data ingestion pipelines.
• Articulate PySpark architecture and performance to stakeholders.
• Implement unit and BDD testing with effective mocking.
• Design and optimise Delta Lake tables for performance.
• Build Docker environments for deployment.
• Integrate Azure services and support CI/CD pipelines.

Skills

Python expertise

PySpark architecture design

Behavior Driven Development (BDD)

Delta Lake optimization

Docker experience

Azure services knowledge

Problem-solving skills

Job description

Job Title: SC Cleared Python Data Engineer - Azure

Contract: 12 months

Location: UK Based - Remote

Start: January 2026

Clearance: Active SC Clearance required

Role Overview

We are seeking an SC Cleared Python Data Engineer with strong hands‑on experience in PySpark, Delta Lake, and Azure cloud services. The role focuses on designing and delivering scalable, well‑tested data pipelines, with particular importance placed on the ability to understand, explain, and design PySpark architectures and demonstrate deep, production‑grade Python expertise.

You will work in a containerised, cloud‑native environment, delivering maintainable, configurable, and test‑driven solutions as part of a multi‑disciplinary engineering team.

Key Responsibilities

• Design, develop, and maintain data ingestion and transformation pipelines using Python and PySpark.
• Clearly articulate PySpark architecture, execution models, and performance considerations to both technical and non‑technical stakeholders.
• Implement unit and BDD testing (Behave or similar), including effective mocking and dependency management.
• Design and optimise Delta Lake tables to support ACID transactions, schema evolution, and incremental processing.
• Build and manage Docker‑based environments for development, testing, and deployment.
• Develop configuration‑driven, reusable codebases suitable for multiple environments.
• Integrate Azure services including Azure Functions, Key Vault, and Blob/Data Lake Storage.
• Optimise Spark jobs for performance, scalability, and reliability in production.
• Collaborate with Cloud, DevOps, and Data teams to support CI/CD pipelines and environment consistency.
• Produce clear technical documentation and follow cloud security and data governance best practices.

Required Skills & Experience

• Strong Python expertise, with a demonstrable depth of experience in designing modular, testable, production‑quality code.
• Proven experience explaining and designing PySpark architectures, including distributed processing and performance tuning.
• Hands‑on experience with Behave or similar BDD frameworks, including mocking and patching techniques.
• Solid understanding of Delta Lake concepts, transactional guarantees, and optimisation strategies.
• Experience using Docker across development and deployment workflows.
• Practical experience with Azure services (Functions, Key Vault, Blob Storage, ADLS Gen2).
• Experience building configuration‑driven applications.
• Strong problem‑solving skills and ability to work independently in agile environments.

Desirable Experience

• Databricks or Synapse with Delta Lake.
• CI/CD pipelines (Azure DevOps or similar) and infrastructure‑as‑code.
• Knowledge of Azure data security and governance best practices.
• Experience working in distributed or multi‑team environments.