Security-Cleared Data Engineer — Azure, Airflow & Spark

iO Associates; Greater London; Hybrid; GBP 89,000 - 148,000; Full time

Job summary

London-based Data Engineer is sought on a contract basis to support a MoD-owned data platform. The role focuses on building and maintaining secure data pipelines using Python, Azure and Kubernetes, with Apache Airflow and Apache Spark integrated into the stack.

You will work within a government-aligned, secure environment, ensuring data remains hosted on MoD infrastructure while collaborating with delivery teams to troubleshoot and improve pipelines.

Qualifications

• Strong commercial experience as a Data Engineer.
• Experience working with Azure.
• Experience with Kubernetes.
• Hands-on experience with Apache Airflow and Apache Spark.
• Experience building and maintaining data pipelines.
• Experience working within government, defence or other secure environments.
• Strong understanding of data engineering principles and best practices.

Responsibilities

• Build and maintain new and existing data pipelines.
• Develop data solutions using Python, Azure and Kubernetes.
• Work with Apache Airflow and Apache Spark.
• Support the ongoing development and improvement of the data platform.
• Ensure data remains securely hosted on MoD-owned infrastructure.
• Collaborate with wider technical and delivery teams.
• Troubleshoot and resolve technical issues across existing data pipelines.

Skills

Data engineering fundamentals

Secure environments experience

Data pipelines

Tools

Python

Azure

Kubernetes

Apache Airflow

Apache Spark

Job description

London-based Data Engineer is sought on a contract basis to support a MoD-owned data platform. The role focuses on building and maintaining secure data pipelines using Python, Azure and Kubernetes, with Apache Airflow and Apache Spark integrated into the stack.

You will work within a government-aligned, secure environment, ensuring data remains hosted on MoD infrastructure while collaborating with delivery teams to troubleshoot and improve pipelines.