Data Engineer

DataCareers; City Of London; On-site; GBP 70,000 - 110,000; Full time

Job summary

DataCareers is recruiting a DV cleared Data Engineer for a leading technology consultancy. You will join an established defence programme and be responsible for building, maintaining and improving data pipelines within a secure MoD environment.

On-site in London 4 days a week, you will work with Python, Azure, Kubernetes, Apache Airflow and Spark to deliver scalable data solutions while ensuring governance and data security in MoD-owned infrastructure.

Qualifications

• Experience building and maintaining production data pipelines.
• Experience delivering data solutions in secure defence or government environments.
• Proficient in Python and Spark-based data processing.
• Experience with Azure, Kubernetes and Apache Airflow.

Responsibilities

• Build and maintain robust data pipelines using Python.
• Work across Azure environments on the low side.
• Support data engineering workloads within Kubernetes environments on the high side.
• Develop and maintain workflows using Apache Airflow.
• Work with Apache Spark for large-scale data processing.
• Support the ongoing development and improvement of established data pipelines.
• Help ensure data remains within MoD-owned infrastructure in line with data governance requirements.
• Work within an existing long-running programme requiring additional delivery capacity.
• Active DV Clearance.

Skills

Python

Data pipelines

Data governance

Tools

Azure

Kubernetes

Apache Airflow

Apache Spark

Palantir

Job description

We are looking to recruit a DV Cleared Data Engineer for a leading technology consultancy. This will be joining an established defence programme requiring additional engineering support across an existing data platform.

You will be responsible for building, maintaining and improving new and existing data pipelines within a secure MoD environment.

Due to the nature of the work you will be required to work on site in London 4 days a week.

Key Responsibilities

• Build and maintain robust data pipelines using Python
• Work across Azure environments on the low side
• Support data engineering workloads within Kubernetes environments on the high side
• Develop and maintain workflows using Apache Airflow
• Work with Apache Spark for large-scale data processing
• Support the ongoing development and improvement of established data pipelines
• Help ensure data remains within MoD-owned infrastructure in line with data governance requirements
• Work within an existing long-running programme requiring additional delivery capacity
• Active DV Clearance
• Experience with Azure
• Strong knowledge of Kubernetes
• Experience with Apache Airflow
• Experience with Apache Spark
• Strong background building and maintaining production data pipelines
• Previous experience working within a Palantir environment
• Experience delivering data solutions within secure defence or government environments