Senior Data Engineer & Analyst — Real-Time Pipelines

Opus Recruitment Solutions Limited; Newcastle upon Tyne; On-site; GBP 50,000 - 75,000; Full time

Job summary

Opus Recruitment Solutions Limited in the United Kingdom is seeking a data engineer to build and optimise data pipelines using Java (preferred) or Python, develop real-time streaming and batch workflows, and collaborate with analytics, AI/ML, product and engineering teams.

This role involves working with Kafka, Flink or Spark, AWS Kinesis/MSK/Lambda/Glue, governance and security controls, and deploying containerised workloads with Docker, Kubernetes and AWS EKS; experience with Terraform and

Qualifications

• Build and optimise data pipelines using Java (preferred) or Python.
• Develop real-time streaming and batch data pipelines.
• Work with Kafka, Flink or Spark, with Kafka/Flink preferred.
• Develop AWS-based data solutions using Kinesis, MSK, Lambda and Glue.
• Implement data governance, lineage, security and quality controls.
• Support CI/CD using Azure DevOps, GitHub Actions or Jenkins.
• Use Terraform and Infrastructure as Code practices.
• Deploy containerised workloads using Docker, Kubernetes and AWS EKS.

Responsibilities

• Build and optimise data pipelines using Java (preferred) or Python.
• Develop real-time streaming and batch data pipelines.
• Work with Kafka, Flink or Spark, with Kafka/Flink preferred.
• Develop AWS-based data solutions using Kinesis, MSK, Lambda and Glue.
• Implement data governance, lineage, security and quality controls.
• Support CI/CD using Azure DevOps, GitHub Actions or Jenkins.
• Use Terraform and Infrastructure as Code practices.
• Deploy containerised workloads using Docker, Kubernetes and AWS EKS.

Skills

Data pipelines

Java

Python

Real-time streaming

Batch processing

Kafka

Flink

Spark

CI/CD

Terraform

Docker

Kubernetes

AWS

Tools

Kafka

Flink

Spark

Kinesis

MSK

Lambda

Glue

Terraform

Docker

Kubernetes

AWS EKS

Azure DevOps

GitHub Actions

Jenkins

Job description

Opus Recruitment Solutions Limited in the United Kingdom is seeking a data engineer to build and optimise data pipelines using Java (preferred) or Python, develop real-time streaming and batch workflows, and collaborate with analytics, AI/ML, product and engineering teams.

This role involves working with Kafka, Flink or Spark, AWS Kinesis/MSK/Lambda/Glue, governance and security controls, and deploying containerised workloads with Docker, Kubernetes and AWS EKS; experience with Terraform and