Data Pipeline Software Developer — Hybrid, AWS & Spark

Scrumconnect Limited; Newcastle upon Tyne; Hybrid; GBP 70,000 - 95,000; Full time

Job summary

Scrumconnect Consulting is seeking an experienced Software Developer to join our engineering team, focusing on data engineering and platform solutions. You will design, build, and maintain data pipelines and infrastructure to support secure, reliable, and scalable public sector services.

You will work with AWS, Terraform, Spark, Airflow, Docker, and GitLab CI/CD within a hybrid model, contributing to secure coding practices and collaborating across cross-functional teams.

Qualifications

• Experience with AWS services and Terraform for infrastructure as code.
• Proficient in Apache Spark for data processing and Apache Airflow for workflow orchestration.
• Experience with Docker and containerisation; CI/CD with GitLab.
• Good understanding of secure coding practices and security-scanning integration.
• Experience with Jupyter Notebooks for data exploration and analysis.
• Experience integrating with third-party or internal platform products, ideally in government or DWP contexts.

Responsibilities

• Design, develop, and maintain data pipelines and processing workflows using Spark and Python.
• Build, schedule, and monitor workflow orchestration with Airflow.
• Develop and maintain infrastructure as code using Terraform across AWS.
• Containerise applications and services with Docker.
• Build and maintain CI/CD pipelines using GitLab for automated builds, tests, deployments.
• Integrate security scanning into the development and deployment pipelines.
• Develop and maintain Jupyter notebooks for data exploration and prototyping.
• Collaborate with data engineers, DevOps, and product stakeholders.
• Write clean, tested code and contribute to system design and architecture decisions.
• Ensure performance, scalability, and security compliance.

Skills

Analytical thinking

Communication skills

Independent work

Mentorship mindset

Tools

AWS

Terraform

Apache Spark

Apache Airflow

Docker

GitLab CI/CD

Security scanning tools

Jupyter Notebooks

Git

Job description

Scrumconnect Consulting is seeking an experienced Software Developer to join our engineering team, focusing on data engineering and platform solutions. You will design, build, and maintain data pipelines and infrastructure to support secure, reliable, and scalable public sector services.

You will work with AWS, Terraform, Spark, Airflow, Docker, and GitLab CI/CD within a hybrid model, contributing to secure coding practices and collaborating across cross-functional teams.