Data Pipeline Engineer

ISR RECRUITMENT LIMITED; United Kingdom; Remote; GBP 65,000 - 101,000; Part time

Benefits offered by this job

Remote work (UK-based)

Job summary

ISR RECRUITMENT LIMITED seeks a Data Pipeline Engineer / Data Pipeline Architect for a central government project. The role is remote (UK-based) and focuses on building an open-source, cloud-agnostic data transformation layer using NiFi, RDF, SPARQL, Python and Java.

You will create reusable pipelines, ensure data quality with testing, and support secure exchange between systems using open-source tools.

Qualifications

• Proven experience designing data integration or ETL solutions with Apache NiFi.
• Experience with RDF triples, SPARQL and semantic data modelling.
• Hands-on Python and Java development experience.
• Experience with APIs, databases, and file data sources.

Responsibilities

• Design and evolve the architecture of a reusable, cloud-agnostic data transformation platform.
• Build, test and maintain Apache NiFi flows, including integrations with APIs, databases, files and messaging systems.
• Develop pipelines to ingest, transform, validate and publish RDF triples, using SPARQL for querying and updates.
• Develop supporting components and integrations using Python and Java.
• Define data mappings, interface contracts, technical specifications and architectural decisions.
• Implement robust processing, including error handling, retries, duplicate management and recovery.
• Establish automated testing, monitoring, version control and repeatable deployment practices.
• Collaborate with architects, developers and stakeholders to translate requirements into working solutions.
• Produce clear technical documentation and support the delivery of reusable platform capabilities.

Skills

Apache NiFi

RDF triples

SPARQL

Python

Java

Data integration

CI/CD

Version control

Tools

Apache Jena

Kafka

SHACL

Docker

Kubernetes

Fuseki

Job description

Data Pipeline Engineer / Data Pipeline Architect

* 3-month contract (initially)

* Remote Working (UK-based)

* Market Rates

* Outside IR35

The Opportunity

We are supporting a central government client who are seeking an experienced Data Pipeline Engineer and Architect to help design and build an open-source Data Transformation Layer for a national data exchange platform.

This is a hands-on engineering and architecture role, combining the design of reusable, reliable data pipelines with practical development using Apache NiFi, RDF, SPARQL, Python and Java.

The platform will use open-source, cloud-agnostic technologies to ingest, transform, validate and exchange data securely between systems.

Skills and Experience

• Proven experience designing and delivering data integration or ETL solutions using Apache NiFi.
• Practical project experience with RDF triples, SPARQL and semantic data modelling.
• Hands-on development experience in both Python and Java.
• Experience designing reusable pipelines across APIs, databases, files and other data sources.
• Understanding of reliable data processing, validation, automated testing and operational monitoring.
• Experience with version control, CI/CD, secure integrations and repeatable deployments.
• Ability to translate high-level requirements into technical designs and production-ready solutions.

Desirable Experience

• Apache Jena and Jena Fuseki.
• Apache Kafka and event-driven data integration.
• SHACL or other RDF validation approaches.
• Custom NiFi processor development.
• Docker, Kubernetes and portable deployment architectures.
• Large RDF datasets, SPARQL optimisation or secure data exchange between organisations.

Key Technologies

Apache NiFi | RDF | SPARQL | Python | Java | Apache Jena | Fuseki | Kafka | ETL | Data Integration | Open Source

Role and Responsibilities

• Design and evolve the architecture of a reusable, cloud-agnostic data transformation platform.
• Build, test and maintain Apache NiFi flows, including integrations with APIs, databases, files and messaging systems.
• Develop pipelines to ingest, transform, validate and publish RDF triples, using SPARQL for querying and updates.
• Develop supporting components and integrations using Python and Java.
• Define data mappings, interface contracts, technical specifications and architectural decisions.
• Implement robust processing, including error handling, retries, duplicate management and recovery.
• Establish automated testing, monitoring, version control and repeatable deployment practices.
• Collaborate with architects, developers and stakeholders to translate requirements into working solutions.
• Produce clear technical documentation and support the delivery of reusable platform capabilities.

NB: Candidates must be eligible for BPSS security clearance which will processed following successful interviews (5-7 days).