DV Cleared Data Engineer – Real-Time Pipelines (On-site)

Searchability NS&D; Bradley Stoke; On-site; GBP 140,000 - 190,000; Full time

Job summary

National Security Organisation in Gloucester is seeking an eDV cleared Data Engineer for an on-site, full-time role. You will design and manage data pipelines, integrate diverse sources, and apply real-time and batch processing at scale.

The ideal candidate has experience with Apache NiFi, Kafka, SQL/NoSQL, and ETL languages (Groovy, Python or Java). DV clearance (West) is essential, with willingness to work on-site as required.

Qualifications

• Active eDV clearance (West) required.
• Willingness to work on-site in Gloucester full-time.
• Experience with data pipelines and ETL in security domain.

Responsibilities

• Design and develop data pipelines, including ingestion, orchestration, and ETL processing (e.g., NiFi).
• Ensure data consistency, quality, and security across all processes.
• Create and maintain database schemas and data models.
• Integrate and enrich data from diverse sources, maintaining data integrity.
• Maintain and enhance Data Ingest and Data Stores.
• Troubleshoot and diagnose issues within integrated (enriched) data systems.
• Collaborate with the scrum team to decompose user requirements into epics and stories.
• Write clean, secure, and reusable code following a test-driven development approach.
• Monitor system performance and implement updates to maintain optimal operation.

Skills

Data pipelines

ETL processing

Data integration

Apache

SQL/NoSQL

Groovy

Python

Java

Docker

Kubernetes

Grafana

Prometheus

Integration / debugging

System architectures

Tools

Apache NiFi

MongoDB

Job description

National Security Organisation in Gloucester is seeking an eDV cleared Data Engineer for an on-site, full-time role. You will design and manage data pipelines, integrate diverse sources, and apply real-time and batch processing at scale.

The ideal candidate has experience with Apache NiFi, Kafka, SQL/NoSQL, and ETL languages (Groovy, Python or Java). DV clearance (West) is essential, with willingness to work on-site as required.