Cloud Data Engineer: Spark, Airflow & AWS | Hybrid UK

Scrumconnect Limited; Newcastle upon Tyne; Hybrid; GBP 70,000 - 95,000; Full time

Job summary

Scrumconnect Limited is seeking a hands-on Data Engineer in the United Kingdom to design, build, and maintain data pipelines on a modern AWS-native stack. You will use Spark/PySpark for distributed processing, Airflow for orchestration, and Terraform for infrastructure as code in a hybrid work arrangement with weekly travel to Newcastle office.

You will contribute to data governance, dimensional modeling, and enterprise-grade data assets while ensuring reliability and security across the data

Qualifications

• Experience building data pipelines on AWS in a cloud-native environment.
• Knowledge of dimensional data models and data governance practices.
• Experience with Spark/PySpark for distributed data processing.
• Hands-on experience with Airflow for workflow orchestration and scheduling.

Responsibilities

• Design, build, and maintain large-scale data pipelines on a cloud platform.
• Develop and manage Airflow DAGs for reliable data workflows.
• Perform root-cause analysis of pipeline failures and data quality issues.
• Implement infrastructure as code using Terraform; containerise with Docker.
• Collaborate with data stakeholders to translate requirements into scalable solutions.

Skills

Data analysis

Root cause analysis

Collaborative teamwork

Problem solving

Tools

PySpark

Apache Spark

Apache Airflow

Terraform

Docker

GitLab

AWS (EMR/S3/CloudWatch)

Job description

Scrumconnect Limited is seeking a hands-on Data Engineer in the United Kingdom to design, build, and maintain data pipelines on a modern AWS-native stack. You will use Spark/PySpark for distributed processing, Airflow for orchestration, and Terraform for infrastructure as code in a hybrid work arrangement with weekly travel to Newcastle office.

You will contribute to data governance, dimensional modeling, and enterprise-grade data assets while ensuring reliability and security across the data