Airflow-Spark-Scala Data Engineer (Hybrid Glasgow)

eTeam; Glasgow; Hybrid; GBP 65,000 - 90,000; Full time

Job summary

eTeam is seeking a data engineer in Glasgow (Hybrid 2/3 days per week) to design, implement and monitor data orchestration workflows using Apache Airflow. You will process large-scale datasets with Python/Scala and Spark, optimize Postgres/Starburst queries, and support CI/CD and containerized deployments in a multi-environment setup.

The role requires collaboration with cross-functional teams, strong data quality focus, and adherence to risk and control standards.

Qualifications

• Experience building data pipelines and workflows with Airflow.
• Strong SQL (PostgreSQL) and data validation skills.
• Exposure to distributed data processing (Spark/Starburst/Trino) and cloud concepts.

Responsibilities

• Develop, maintain and troubleshoot data orchestration workflows using Apache Airflow for extract generation, data publishing, feed delivery and batch processes.
• Build and support reusable engineering components using Python, Scala and Spark to process and validate large finance datasets.
• Design and optimize SQL queries for metadata processing, data validation and operational investigation.
• Support production issues including failed DAGs, missing files and data mismatches; ensure reliable delivery.
• Contribute to platform engineering and release activity using GitLab, Docker and CI/CD across DEV/UAT/production-like environments.
• Enhance monitoring, logging, alerting and automation to improve stability and control evidence.
• Collaborate with application, platform, database, infrastructure and downstream teams to diagnose issues end-to-end and implement safe fixes.
• Maintain risk and control mindset, ensuring changes are tested, reviewed and released under Barclays standards.
• Experience with Starburst/Trino, AWS S3 and distributed query behavior for large data validation and troubleshooting
• Familiarity with React JS or API-driven UIs for configuration and monitoring workflows
• Exposure to OpenShift, Kubernetes, Helm in enterprise deployments.

Skills

Data engineering

ETL design

Troubleshooting

Collaboration

Risk awareness

Tools

Apache Airflow

Python

Scala

Spark

PostgreSQL

Starburst/Trino

GitLab

Docker

CI/CD

OpenShift

Kubernetes

Helm

ReactJS

AWS S3

Job description

eTeam is seeking a data engineer in Glasgow (Hybrid 2/3 days per week) to design, implement and monitor data orchestration workflows using Apache Airflow. You will process large-scale datasets with Python/Scala and Spark, optimize Postgres/Starburst queries, and support CI/CD and containerized deployments in a multi-environment setup.

The role requires collaboration with cross-functional teams, strong data quality focus, and adherence to risk and control standards.