Data Engineer: Spark, Airflow & Financial Pipelines Hybrid

HelloKindred; Scotland; Hybrid; GBP 70,000 - 110,000; Full time

Job summary

HelloKindred, a specialist staffing partner for marketing, creative and technology roles, seeks a Data Engineer in a hybrid setup based in the United Kingdom. The role focuses on developing and maintaining data orchestration workflows, processing large financial datasets, and ensuring platform reliability with strong emphasis on security and change controls.

Key responsibilities include building reusable components, optimizing SQL and Starburst/Trino queries, and collaborating across

Qualifications

• Strong Spark, Scala, and Python data engineering experience.
• Experience building and maintaining Apache Airflow workflows.
• SQL with PostgreSQL and data validation/reconciliation.
• Experience across production data workflows and troubleshooting.

Responsibilities

• Develop, maintain, and troubleshoot data orchestration workflows using Apache Airflow for extract generation, data publishing, feed delivery, and operational batch processes.
• Build and support reusable engineering components using Python, Scala, and Sharedview Spark to process, transform, and validate large-scale financial datasets.
• Design, write, and optimize SQL/PostgreSQL and Starburst/Trino queries for metadata processing, data validation, reconciliation, and operational investigation.
• Support Output Gateway production issues, including failed DAGs, delayed extracts, missing files, data mismatches, checksum failures, logs, status updates, and downstream delivery exceptions.
• Contribute to platform engineering and release activities using GitLab, Docker, and controlled CI/CD practices across DEV, UAT, and production-like environments.
• Enhance monitoring, logging, alerting, audit tracking, and automation to improve operational stability, supportability, and evidence for control processes.
• Work with application, platform, database, infrastructure, and downstream teams to diagnose issues end-to-end and implement safe, controlled fixes.
• Maintain a strong risk and control mindset by ensuring changes are tested, reviewed, documented, and released in accordance with technology, security, and change standards.

Skills

Spark

Scala

Python

Airflow

SQL

PostgreSQL

Starburst/Trino

GitLab

Docker

CI/CD

Monitoring

Collaboration

Risk mindset

Tools

GitLab

Docker

Kubernetes

OpenShift

Helm

Job description

HelloKindred, a specialist staffing partner for marketing, creative and technology roles, seeks a Data Engineer in a hybrid setup based in the United Kingdom. The role focuses on developing and maintaining data orchestration workflows, processing large financial datasets, and ensuring platform reliability with strong emphasis on security and change controls.

Key responsibilities include building reusable components, optimizing SQL and Starburst/Trino queries, and collaborating across