Hybrid Data Engineer: Spark, Airflow & Financial Data Pipelines

HelloKindred; Glasgow; Hybrid; GBP 60,000 - 90,000; Full time

Job summary

HelloKindred in the United Kingdom seeks a Data Engineer to design, develop, and maintain data pipelines supporting large financial datasets. You will work across application, platform, database, infrastructure, and downstream teams to ensure reliable data processes and compliance with security and change standards.

The role emphasizes Spark/Scala/Python, Airflow orchestration, SQL optimization, and robust CI/CD practices using GitLab, Docker, and Starburst/Trino knowledge.

Qualifications

• Strong skills and experience in data engineering using Spark, Scala, and Python.
• Experience developing, maintaining, and troubleshooting Apache Airflow workflows for data orchestration and operational batch processes.
• Hands-on experience building reusable engineering components to process, transform, and validate large-scale datasets.
• Proficiency in SQL and PostgreSQL, including query design, optimization, metadata processing, data validation, and reconciliation.
• Experience supporting production data workflows and troubleshooting failed DAGs, delayed extracts, missing files, data mismatches, checksum failures, and downstream delivery exceptions.
• Familiarity with GitLab, Docker, and controlled CI/CD practices across development, UAT, and production-like environments.
• Understanding of monitoring, logging, alerting, audit tracking, and automation to improve operational stability and supportability.
• Ability to collaborate with teams to diagnose and resolve technical issues.
• Strong awareness of risk, controls, testing, documentation, and technology security standards.

Responsibilities

• Develop, maintain, and troubleshoot data orchestration workflows using Apache Airflow for extract generation and data delivery.
• Build and support reusable components using Python/Scala to process and validate large-scale financial datasets.
• Design and optimize PostgreSQL/SQL queries for metadata processing and data validation.
• Support production data workflows, troubleshoot DAGs, missing files, and delivery exceptions.
• Contribute to platform engineering and release activities using GitLab, Docker, and CI/CD across DEV/UAT/production-like environments.
• Enhance monitoring, logging, and alerting to improve operational stability and auditability.
• Collaborate with cross-functional teams to diagnose issues and implement safe fixes.
• Maintain risk and control mindset with tested, documented, and approved changes.

Skills

Spark

Scala

Python

Airflow

SQL

PostgreSQL

GitLab

Docker

CI/CD

Kubernetes

Monitoring

Data validation

Tools

Starburst/Trino

AWS S3

OpenShift

Job description

HelloKindred in the United Kingdom seeks a Data Engineer to design, develop, and maintain data pipelines supporting large financial datasets. You will work across application, platform, database, infrastructure, and downstream teams to ensure reliable data processes and compliance with security and change standards.

The role emphasizes Spark/Scala/Python, Airflow orchestration, SQL optimization, and robust CI/CD practices using GitLab, Docker, and Starburst/Trino knowledge.