Data skills (Spark + Scala)

eTeam; Glasgow; Hybrid; GBP 65,000 - 90,000; Full time

Job summary

eTeam is seeking a data engineer in Glasgow (Hybrid 2/3 days per week) to design, implement and monitor data orchestration workflows using Apache Airflow. You will process large-scale datasets with Python/Scala and Spark, optimize Postgres/Starburst queries, and support CI/CD and containerized deployments in a multi-environment setup.

The role requires collaboration with cross-functional teams, strong data quality focus, and adherence to risk and control standards.

Qualifications

• Experience building data pipelines and workflows with Airflow.
• Strong SQL (PostgreSQL) and data validation skills.
• Exposure to distributed data processing (Spark/Starburst/Trino) and cloud concepts.

Responsibilities

• Develop, maintain and troubleshoot data orchestration workflows using Apache Airflow for extract generation, data publishing, feed delivery and batch processes.
• Build and support reusable engineering components using Python, Scala and Spark to process and validate large finance datasets.
• Design and optimize SQL queries for metadata processing, data validation and operational investigation.
• Support production issues including failed DAGs, missing files and data mismatches; ensure reliable delivery.
• Contribute to platform engineering and release activity using GitLab, Docker and CI/CD across DEV/UAT/production-like environments.
• Enhance monitoring, logging, alerting and automation to improve stability and control evidence.
• Collaborate with application, platform, database, infrastructure and downstream teams to diagnose issues end-to-end and implement safe fixes.
• Maintain risk and control mindset, ensuring changes are tested, reviewed and released under Barclays standards.
• Experience with Starburst/Trino, AWS S3 and distributed query behavior for large data validation and troubleshooting
• Familiarity with React JS or API-driven UIs for configuration and monitoring workflows
• Exposure to OpenShift, Kubernetes, Helm in enterprise deployments.

Skills

Data engineering

ETL design

Troubleshooting

Collaboration

Risk awareness

Tools

Apache Airflow

Python

Scala

Spark

PostgreSQL

Starburst/Trino

GitLab

Docker

CI/CD

OpenShift

Kubernetes

Helm

ReactJS

AWS S3

Job description

We are a Global Recruitment specialist that provides support to the clients across EMEA, APAC, US and Canada. We have an excellent job opportunity for you

Location: Glasgow (Hybrid 2/3 days per week)

Duration: 31/12/2026

Role Description

JD

Key Accountabilities

• Develop, maintain and troubleshoot data orchestration workflows using Apache Airflow for extract generation, data publishing, feed delivery and operational batch processes.
• Build and support reusable engineering components using Python, Scala and Sharedview Spark to process, transform and validate large-scale finance datasets.
• Design, write and optimise SQL/PostgresQL and Starburst/Trino queries for metadata processing, data validation, reconciliation and operational investigation
• Support Output Gateway production issues including failed DAGs, delayed extracts missing files, data mismatches, checksum failures, logs, status updates and downstream delivery exceptions
• Contribute to platform engineering and release activity using GitLab, Docker and controlled CI/CD practices across DEV, UAT and production-like environments
• Enhance monitoring, logging, alerting, audit tracking and automation to improve operational stability, supportability and evidence for control processes
• Work with application, platform, database, infrastructure and downstream teams to diagnose issues end-to-end and implement safe, controlled fixes
• Maintain a strong risk and control mindset by ensuring changes are tested reviewed, documented and released under Barclays technology, security and change standards.
• Experience with Starburst/Trino, AWS S3 concepts and distributed query behavior for large dataset validation and troubleshooting
• Exposure to React JS or API-driven operational user interfaces for configuration, monitoring and support workflows
• Familiarity with OpenShift, Kubernetes, Helm or container platform deployments ir enterprise environments.

Understanding of financial data controls, auditability, reconciliation, lineage operational resilience and data quality expectations.