Security-Cleared Data Engineer: Real-Time Pipelines

Capgemini; Greater London; Hybrid; GBP 60,000 - 90,000; Full time

Benefits offered by this job

Hybrid working

Learning for life – 250,000 courses

Flexible working arrangements

Job summary

Capgemini is seeking a code‑first Data Engineer to design, build and operate scalable data pipelines for analytics, AI, GenAI and agentic systems. You’ll ingest, transform and serve data, govern datasets, and enable secure, auditable data services in defence contexts.

You will work across the full lifecycle from development to production, deploying in secure containerised environments, and contributing to CI/CD.

Qualifications

• Expert‑level Python and strong software engineering foundations
• Building pipelines with streaming frameworks and distributed processing engines
• Relational or analytical storage experience (e.g., PostgreSQL)
• Orchestration of data workflows with Airflow, Dagster or Prefect
• Data quality and lineage tooling (Great Expectations, dbt)
• AI/MLops knowledge including GenAI and agentic systems
• Containerisation and Kubernetes for secure environments
• Hands‑on CI/CD and production data pipelines in Defence/secure contexts

Responsibilities

• Develop modular software components for data processing and analytics with testing and observability
• Design and implement real‑time data ingestion and event‑driven patterns
• Define and run data workflows with modern orchestration frameworks
• Package, deploy and monitor ML and GenAI workloads in secure environments
• Containerise services and deploy into secure Kubernetes or cloud environments
• Implement health monitoring, logging, metrics and data lineage for auditability

Skills

Python

Software engineering

Data pipelines

Kafka

Spark

PostgreSQL

Airflow

Dagster

Prefect

Great Expectations

dbt

Docker

Kubernetes

CI/CD

MLOps / GenAI

Tools

Kafka

Spark

PostgreSQL

Airflow

Dagster

Prefect

Great Expectations

dbt

Docker

Kubernetes

GitHub Actions

Job description

Capgemini is seeking a code‑first Data Engineer to design, build and operate scalable data pipelines for analytics, AI, GenAI and agentic systems. You’ll ingest, transform and serve data, govern datasets, and enable secure, auditable data services in defence contexts.

You will work across the full lifecycle from development to production, deploying in secure containerised environments, and contributing to CI/CD.