Senior Data Engineer: Real-Time Pipelines in Elastic Stack

Advanced Resource Managers; United Kingdom; On-site; GBP 60,000 - 80,000; Full time

Job summary

A prominent recruitment agency is seeking a Senior Data Engineer for a 12-month contract in Bristol. The role involves designing and managing data pipelines using the Elastic Stack and Apache NiFi within a large aerospace company. Candidates should have experience in regulated industries and familiarity with ETL tools and containerization. This position offers competitive hourly pay and requires skills in ensuring data quality, compliance, and developing dashboards for informed decision-making.

Responsibilities

• Design, develop, and maintain secure and scalable data pipelines using Elastic Stack.
• Implement data ingestion, transformation, and integration processes.
• Collaborate to ensure compliance with security policies.
• Manage and monitor large-scale data flows in real-time.
• Develop robust data models for analytics and reporting.
• Troubleshoot, debug, and tune performance of data pipelines.
• Build dashboards in Kibana for data-driven decisions.
• Ensure high availability and disaster recovery for data systems.
• Document data architecture and workflows for operational handover.

Skills

Experience in government, defence, or highly regulated industries

ETL tools like Apache Kafka, Spark, or Hadoop

Containerization and orchestration tools such as Docker and Kubernetes

Monitoring and alerting tools such as Prometheus, Grafana, or ELK

Understanding of ML algorithms

Confidence in developing end-to-end solutions

Infrastructure as code (Terraform, Ansible)

Job description

A prominent recruitment agency is seeking a Senior Data Engineer for a 12-month contract in Bristol. The role involves designing and managing data pipelines using the Elastic Stack and Apache NiFi within a large aerospace company. Candidates should have experience in regulated industries and familiarity with ETL tools and containerization. This position offers competitive hourly pay and requires skills in ensuring data quality, compliance, and developing dashboards for informed decision-making.