Data Engineer - Spark & Scala

GIOS Technology; Glasgow; Hybrid; GBP 60,000 - 90,000; Full time

Job summary

GIOS Technology Limited in Glasgow, UK, is seeking a Data Engineer specialized in Spark and Scala to design, build and maintain large-scale data pipelines. The role operates in a hybrid model (2–3 days onsite) and focuses on data processing, validation and orchestration using Airflow.

The ideal candidate will be proficient in Python, SQL and PostgreSQL, with production support experience and a strong focus on data quality, monitoring and resilient pipelines.

Qualifications

• Hands-on Spark and Scala for large-scale data processing.
• Experience building data pipelines and orchestration using Airflow.
• Proficiency in Python, SQL, and PostgreSQL for data processing and validation.
• Experience with production support and debugging failed workflows.
• Familiarity with GitLab, CI/CD, Docker and release practices.
• Understanding of data quality, monitoring and operational resilience.
• Nice to have: Starburst/Trino, AWS S3, Kubernetes or OpenShift.

Responsibilities

• Develop and maintain data pipelines and orchestration workflows using Airflow.
• Transform and validate large-scale datasets with Spark and Scala.
• Process data with Python, SQL, and PostgreSQL; troubleshoot issues.
• Investigate production incidents, data mismatches and missing files.
• Collaborate with teams using GitLab, CI/CD, Docker and release practices.
• Ensure data quality, monitoring, logging, auditability and resilience.
• Familiarity with Starburst/Trino, AWS S3, Kubernetes or OpenShift is a plus.

Skills

Spark

Scala

Python

Airflow

SQL

PostgreSQL

Data Engineering

Production Support

Tools

GitLab

CI/CD

Docker

Kubernetes

OpenShift

AWS S3

Starburst/Trino

Job description

We are hiring: Data Engineer – Spark & Scala

Location: Glasgow, UK – Hybrid (2–3 days onsite)

• Strong hands-on experience with Apache Spark and Scala for processing, transforming and validating large-scale datasets.
• Experience developing and maintaining data pipelines and orchestration workflows using Apache Airflow.
• Proficiency in Python, SQL and PostgreSQL for data processing, validation, reconciliation and troubleshooting.
• Experience with production support, investigating failed workflows, data mismatches, missing files and pipeline failures.
• Familiarity with GitLab, CI/CD, Docker and controlled software release practices.
• Understanding of data quality, monitoring, logging, auditability and operational resilience.
• Experience with Starburst/Trino, AWS S3, Kubernetes or OpenShift would be an advantage.

Key Skills: Apache Spark / Scala / Python / Apache Airflow / SQL / PostgreSQL / Starburst / Trino / GitLab / CI/CD / Docker / Data Engineering / Production Support