Data Engineer OUTSIDE IR35

LHH; United Kingdom; On-site; GBP 60,000 - 80,000; Part time

Job summary

A leading recruitment firm is looking for a Defence Lead in the United Kingdom to manage data pipelines and ensure compliance with security standards. The role requires expertise in the Elastic Stack and Apache NiFi, with responsibilities including designing, implementing, and maintaining secure data solutions. Candidates must have experience in regulated industries and be capable of developing end-to-end solutions. This is a contract position offering onsite work with options for compressed hours.

Qualifications

• Experience in government, defence, or regulated industries is essential.
• Familiarity with machine learning algorithms and their implementation.
• Confidence in developing secure data solutions.

Responsibilities

• Design and maintain secure data pipelines using the Elastic Stack.
• Collaborate with security teams for compliance with policies.
• Manage real-time data flows to ensure performance.

Skills

Data pipeline management

Elastic Stack (Elasticsearch, Logstash, Kibana)

Apache NiFi

Data quality and security

End-to-end solution development

Infrastructure as code

Container orchestration (Docker, Kubernetes)

ETL tools (Kafka, Spark, Hadoop)

Monitoring and alerting tools

Tools

Terraform

Ansible

Prometheus

Grafana

Job description

Defence Lead for LHH & Recruitment solutions- recruitment on demand , SOW, Managed Recruitment/RPO. Attraction and Assessment.

MUST HOLD DV CLEARANCE OUTSIDE IR35

This role requires strong expertise in building and managing data pipelines using the Elastic Stack (Elasticsearch, Logstash, Kibana) and Apache NiFi. The successful candidate will design, implement, and maintain scalable, secure data solutions, ensuring compliance with strict security standards and regulations. This is a UK based onsite role with the option of compressed hours.

• Design, develop, and maintain secure and scalable data pipelines using the Elastic Stack (Elasticsearch, Logstash, Kibana) and Apache NiFi.
• Implement data ingestion, transformation, and integration processes, ensuring data quality and security.
• Collaborate with data architects and security teams to ensure compliance with security policies and data governance standards.
• Manage and monitor large-scale data flows in real-time, ensuring system performance, reliability, and data integrity.
• Develop robust data models to support analytics and reporting within secure environments.
• Perform troubleshooting, debugging, and performance tuning of data pipelines and the Elastic Stack.
• Build dashboards and visualizations in Kibana to enable data-driven decision-making.
• Ensure high availability and disaster recovery for data systems, implementing appropriate backup and replication strategies.
• Document data architecture, workflows, and security protocols to ensure smooth operational handover and audit readiness.

Qualifications

• Experience working in government, defence, or highly regulated industries.
• Experience with additional data processing and ETL tools like Apache Kafka, Spark, or Hadoop.
• Familiarity with containerization and orchestration tools such as Docker and Kubernetes.
• Experience with monitoring and alerting tools such as Prometheus, Grafana, or ELK for data infrastructure.
• Understanding of ML algorithms, their development and implementation.
• Confidence developing end-to-end solutions.
• Experience with infrastructure as code e.g. Terraform, Ansible.

Seniority level

• Mid-Senior level

Employment type

• Contract

Job function & Industries

• Information Technology and Analyst
• Defense and Space Manufacturing and Space Research and Technology