Senior Data Engineer

Searchability NS&D; Portsmouth; Hybrid; GBP 81,000 - 99,000; Full time

Job summary

Searchability NS&D in Portsmouth seeks an experienced Databricks Data Engineer to design and build secure, production‑grade data applications supporting defence environments. This hands‑on role sits at the intersection of Software and Data Engineering, with on‑site work three days per week.

You’ll own scalable data pipelines, streaming and batch processing, and work with Python, Kafka, Spark, Docker, Kubernetes, Airflow, and IaC tools to deliver reliable data platforms with governance and

Qualifications

• Strong Python development and software engineering experience.
• Experience building scalable data pipelines and distributed data platforms.
• Knowledge of Kafka, Spark and modern data processing technologies.
• Experience with orchestration tools such as Airflow, Dagster or Prefect.
• Hands‑on experience with Docker, Kubernetes and containerised deployments.
• CI/CD experience with a focus on production ownership and reliability.
• Understanding of data quality, testing and lineage best practices.
• MLOps or production machine learning experience.
• Terraform, Ansible or other Infrastructure as Code tools.
• PostgreSQL and modern data platforms.
• Experience within Defence, National Security or regulated environments.
• Familiarity with air‑gapped, restricted or secure networks.
• Knowledge of Google Distributed Cloud (GDC) or edge cloud technologies.

Responsibilities

• Building scalable data applications and pipelines using Python.
• Developing streaming, event-driven and orchestrated data solutions.
• Deploying and supporting containerised services in Kubernetes environments.
• Implementing observability, monitoring and data governance best practices.
• Leveraging Infrastructure as Code and CI/CD for secure deployments.
• Owning solutions from design and development through to production support.
• Delivering resilient data platforms within secure Defence environments.

Skills

Python development

Data engineering

CI/CD

Data governance

MLOps

Security/compliance

Tools

Docker

Kubernetes

Airflow

Dagster

Prefect

Kafka

Spark

Terraform

Ansible

PostgreSQL

Google Distributed Cloud (GDC)

Job description

• Location: Portsmouth customer site – 3 days p/week on site
• Salary: Up to £90k DoE plus bonus

We’re working with a leading technology consultancy delivering secure, mission critical digital capabilities across Secure Sector programmes. They’re looking for an experienced Databricks Data Engineer to design and build scalable, resilient data applications that support operational users, analytics platforms, and AI-enabled capabilities.

This is a hands‑on technical role sitting at the intersection of Software Engineering and Data Engineering, focused on building secure, production‑grade data services that operate reliably within highly governed Defence environments.

What you’ll be doing:

• Building scalable data applications and pipelines using Python
• Developing streaming, event-driven and orchestrated data solutions
• Deploying and supporting containerised services in Kubernetes environments
• Implementing observability, monitoring and data governance best practices
• Leveraging Infrastructure as Code and CI/CD for secure deployments
• Owning solutions from design and development through to production support
• Delivering resilient data platforms within secure Defence environments

What we’re looking for:

• Strong Python development and software engineering experience
• Experience building scalable data pipelines and distributed data platforms
• Knowledge of Kafka, Spark and modern data processing technologies
• Experience with orchestration tools such as Airflow, Dagster or Prefect
• Hands‑on experience with Docker, Kubernetes and containerised deployments
• CI/CD experience with a focus on production ownership and reliability
• Understanding of data quality, testing and lineage best practices
• MLOps or production machine learning experience
• Terraform, Ansible or other Infrastructure as Code tools
• PostgreSQL and modern data platforms
• Experience within Defence, National Security or regulated environments
• Familiarity with air‑gapped, restricted or secure networks
• Knowledge of Google Distributed Cloud (GDC) or edge cloud technologies

Security Clearance:

Due to the nature of the work, candidates must hold an active SC Clearance and be a British passport holder.