Data Engineer - Senior Analyst

Opus Recruitment Solutions Limited; Newcastle upon Tyne; On-site; GBP 50,000 - 75,000; Full time

Job summary

Opus Recruitment Solutions Limited in the United Kingdom is seeking a data engineer to build and optimise data pipelines using Java (preferred) or Python, develop real-time streaming and batch workflows, and collaborate with analytics, AI/ML, product and engineering teams.

This role involves working with Kafka, Flink or Spark, AWS Kinesis/MSK/Lambda/Glue, governance and security controls, and deploying containerised workloads with Docker, Kubernetes and AWS EKS; experience with Terraform and

Qualifications

• Build and optimise data pipelines using Java (preferred) or Python.
• Develop real-time streaming and batch data pipelines.
• Work with Kafka, Flink or Spark, with Kafka/Flink preferred.
• Develop AWS-based data solutions using Kinesis, MSK, Lambda and Glue.
• Implement data governance, lineage, security and quality controls.
• Support CI/CD using Azure DevOps, GitHub Actions or Jenkins.
• Use Terraform and Infrastructure as Code practices.
• Deploy containerised workloads using Docker, Kubernetes and AWS EKS.

Responsibilities

• Build and optimise data pipelines using Java (preferred) or Python.
• Develop real-time streaming and batch data pipelines.
• Work with Kafka, Flink or Spark, with Kafka/Flink preferred.
• Develop AWS-based data solutions using Kinesis, MSK, Lambda and Glue.
• Implement data governance, lineage, security and quality controls.
• Support CI/CD using Azure DevOps, GitHub Actions or Jenkins.
• Use Terraform and Infrastructure as Code practices.
• Deploy containerised workloads using Docker, Kubernetes and AWS EKS.

Skills

Data pipelines

Java

Python

Real-time streaming

Batch processing

Kafka

Flink

Spark

CI/CD

Terraform

Docker

Kubernetes

AWS

Tools

Kafka

Flink

Spark

Kinesis

MSK

Lambda

Glue

Terraform

Docker

Kubernetes

AWS EKS

Azure DevOps

GitHub Actions

Jenkins

Job description

• Build and optimise data pipelines using Java (preferred) or Python.
• Develop real-time streaming and batch data pipelines.
• Work with Kafka, Flink or Spark, with Kafka/Flink preferred.
• Develop AWS-based data solutions using Kinesis, MSK, Lambda and Glue.
• Implement data governance, lineage, security and quality controls.
• Support CI/CD using Azure DevOps, GitHub Actions or Jenkins.
• Use Terraform and Infrastructure as Code practices.
• Deploy containerised workloads using Docker, Kubernetes and AWS EKS.
• Collaborate with analytics, AI/ML, product and engineering teams.