Senior Data Engineer

Searchability NS&D; Portsmouth; On-site; GBP 54,000 - 90,000; Full time

Job summary

Searchability NS&D is seeking an experienced Databricks Data Engineer to design and build scalable, secure data applications within Defence environments. Portsmouth customer site with 3 days on-site per week, salary up to £90k DoE plus bonus.

You will work at the intersection of Software and Data Engineering, delivering production‑grade data services, observability, CI/CD and IaC within highly governed networks.

Qualifications

• Strong Python development and software engineering experience.
• Experience building scalable data pipelines and distributed data platforms.
• Knowledge of Kafka, Spark and modern data processing technologies.
• Experience with orchestration tools like Airflow, Dagster or Prefect.
• Hands-on experience with containerised deployments (Docker/Kubernetes).
• CI/CD with production ownership and reliability focus.
• Understanding of data quality, testing and lineage.
• MLOps or production ML experience.
• Infrastructure as Code using Terraform or Ansible.
• PostgreSQL and modern data platforms.
• Experience in Defence, National Security or regulated environments.
• Familiarity with air-gapped, restricted or secure networks.
• Knowledge of Google Distributed Cloud or edge cloud technologies.

Responsibilities

• Building scalable data applications and pipelines using Python
• Developing streaming, event-driven and orchestrated data solutions
• Deploying and supporting containerised services in Kubernetes environments
• Implementing observability, monitoring and data governance best practices
• Leveraging Infrastructure as Code and CI/CD for secure deployments
• Owning solutions from design and development through to production support
• Delivering resilient data platforms within secure Defence environments

Skills

Python

Data pipelines

Kafka

Spark

Airflow

Dagster

Prefect

Docker

Kubernetes

CI/CD

Data quality

Data lineage

MLOps

Terraform

Ansible

PostgreSQL

Defence environment knowledge

Air-gapped networks

GDC / edge cloud

Tools

Docker

Kubernetes

Terraform

Ansible

Job description

• Location: Portsmouth customer site – 3 days p/week on site
• Salary: Up to £90k DoE plus bonus

We’re working with a leading technology consultancy delivering secure, mission critical digital capabilities across Secure Sector programmes. They’re looking for an experienced Databricks Data Engineer to design and build scalable, resilient data applications that support operational users, analytics platforms, and AI-enabled capabilities.

This is a hands‑on technical role sitting at the intersection of Software Engineering and Data Engineering, focused on building secure, production‑grade data services that operate reliably within highly governed Defence environments.

What you’ll be doing:

• Building scalable data applications and pipelines using Python
• Developing streaming, event-driven and orchestrated data solutions
• Deploying and supporting containerised services in Kubernetes environments
• Implementing observability, monitoring and data governance best practices
• Leveraging Infrastructure as Code and CI/CD for secure deployments
• Owning solutions from design and development through to production support
• Delivering resilient data platforms within secure Defence environments

What we’re looking for:

• Strong Python development and software engineering experience
• Experience building scalable data pipelines and distributed data platforms
• Knowledge of Kafka, Spark and modern data processing technologies
• Experience with orchestration tools such as Airflow, Dagster or Prefect
• Hands‑on experience with Docker, Kubernetes and containerised deployments
• CI/CD experience with a focus on production ownership and reliability
• Understanding of data quality, testing and lineage best practices
• MLOps or production machine learning experience
• Terraform, Ansible or other Infrastructure as Code tools
• PostgreSQL and modern data platforms
• Experience within Defence, National Security or regulated environments
• Familiarity with air‑gapped, restricted or secure networks
• Knowledge of Google Distributed Cloud (GDC) or edge cloud technologies

Security Clearance:

Due to the nature of the work, candidates must hold an active SC Clearance and be a British passport holder.