Senior Data Engineer – Secure Defence Data Platform

Searchability NS&D; Portsmouth; Hybrid; GBP 81,000 - 99,000; Full time

Job summary

Searchability NS&D in Portsmouth seeks an experienced Databricks Data Engineer to design and build secure, production‑grade data applications supporting defence environments. This hands‑on role sits at the intersection of Software and Data Engineering, with on‑site work three days per week.

You’ll own scalable data pipelines, streaming and batch processing, and work with Python, Kafka, Spark, Docker, Kubernetes, Airflow, and IaC tools to deliver reliable data platforms with governance and

Qualifications

• Strong Python development and software engineering experience.
• Experience building scalable data pipelines and distributed data platforms.
• Knowledge of Kafka, Spark and modern data processing technologies.
• Experience with orchestration tools such as Airflow, Dagster or Prefect.
• Hands‑on experience with Docker, Kubernetes and containerised deployments.
• CI/CD experience with a focus on production ownership and reliability.
• Understanding of data quality, testing and lineage best practices.
• MLOps or production machine learning experience.
• Terraform, Ansible or other Infrastructure as Code tools.
• PostgreSQL and modern data platforms.
• Experience within Defence, National Security or regulated environments.
• Familiarity with air‑gapped, restricted or secure networks.
• Knowledge of Google Distributed Cloud (GDC) or edge cloud technologies.

Responsibilities

• Building scalable data applications and pipelines using Python.
• Developing streaming, event-driven and orchestrated data solutions.
• Deploying and supporting containerised services in Kubernetes environments.
• Implementing observability, monitoring and data governance best practices.
• Leveraging Infrastructure as Code and CI/CD for secure deployments.
• Owning solutions from design and development through to production support.
• Delivering resilient data platforms within secure Defence environments.

Skills

Python development

Data engineering

CI/CD

Data governance

MLOps

Security/compliance

Tools

Docker

Kubernetes

Airflow

Dagster

Prefect

Kafka

Spark

Terraform

Ansible

PostgreSQL

Google Distributed Cloud (GDC)

Job description

Searchability NS&D in Portsmouth seeks an experienced Databricks Data Engineer to design and build secure, production‑grade data applications supporting defence environments. This hands‑on role sits at the intersection of Software and Data Engineering, with on‑site work three days per week.

You’ll own scalable data pipelines, streaming and batch processing, and work with Python, Kafka, Spark, Docker, Kubernetes, Airflow, and IaC tools to deliver reliable data platforms with governance and