Cloud Data Engineer: Spark, Airflow & AWS | Hybrid

Scrumconnect Consulting; Newcastle upon Tyne; Hybrid; GBP 39,000 - 65,000; Full time

Job summary

Scrumconnect is seeking a Data Engineer to design, build and maintain data pipelines on a modern AWS-native stack. You will use Apache Spark and PySpark for distributed processing, Apache Airflow for orchestration, and several AWS services to store and analyze data.

You will also implement infrastructure as code with Terraform and ensure data governance and security. The role requires translating customer needs into scalable, reliable data assets, and collaborating with cross-functional teams in

Qualifications

• Hands-on data engineering role building data pipelines on AWS cloud.
• Experience with Spark/PySpark and Airflow for orchestration.
• Familiarity with Terraform, Docker and GitLab CI/CD practices.

Responsibilities

• Data pipeline development: Build and maintain scalable data pipelines using Apache Spark and PySpark.
• Workflow orchestration: Configure and manage Airflow DAGs for reliable scheduling.
• Root cause analysis: Identify and resolve data quality and pipeline issues.
• Data modelling: Apply dimensional models and SCD concepts for trusted assets.
• Infrastructure as code: Provision cloud infra with Terraform; containerise with Docker.
• Security & governance: Work within IAM and encryption standards in AWS.

Skills

Python

SQL

PySpark

Jupyter Notebooks

Tools

Apache Spark

Apache Airflow

Terraform

Docker

GitLab

Job description

Scrumconnect is seeking a Data Engineer to design, build and maintain data pipelines on a modern AWS-native stack. You will use Apache Spark and PySpark for distributed processing, Apache Airflow for orchestration, and several AWS services to store and analyze data.

You will also implement infrastructure as code with Terraform and ensure data governance and security. The role requires translating customer needs into scalable, reliable data assets, and collaborating with cross-functional teams in