Senior Data Engineer (0836ED0)

Referment; Greater London; In loco; GBP 75.000 - 110.000; Tempo pieno

Descrizione del lavoro

Referment is building a fast-growing fintech and is hiring a Data Engineer to own and evolve the ingestion platform end-to-end. You’ll work with AWS streaming, Snowflake, and dbt to deliver reliable pipelines with privacy and governance baked in.

You will implement Snowpark Python procedures, manage schemas, and extend Terraform modules across environments, while building observability with Datadog and GitHub Actions for CI/CD.

Competenze

• Hands-on experience with streaming and event ingestion on AWS (Kinesis, EventBridge) and S3 partitioning.
• Strong SQL skills on Snowflake including Snowpipe and Snowpark workloads.
• Proficient in Python for data pipelines and dbt modeling with testing.
• Experience with Terraform/Terragrunt across multiple environments.
• Familiar with secrets management, PII and data governance practices.

Mansioni

• Own and evolve the event ingestion path from EventBridge to Firehose to S3, tuning throughput.
• Design and deploy Snowpark-based Python procedures for data loads and Snowflake Tasks.
• Collaborate with product engineers on upstream schema changes and late-arriving data.
• Extend Terraform/Terragrunt modules for ingestion infra across environments.
• Create Datadog monitors for ingestion components and triage failures.
• Ensure privacy-first pipelines with masking and audit-ready data in Snowflake.

Conoscenze

Streaming ingestion

Python data pipelines

Snowflake SQL

dbt modeling

Terraform / Terragrunt

Kinesis / EventBridge

S3 partitioning

Data governance

PII handling

GitHub Actions

Strumenti

Terraform

Terragrunt

Snowflake

Snowpipe

Snowpark

Kinesis

EventBridge

S3

dbt

Datadog

Descrizione del lavoro

Referment is working with a fast-growing fintech that is building a zero-fee payments platform for merchants, funded by well-known venture investors and already processing at a significant run rate with millions of customers.

You'll join a data team that runs an event-driven platform on AWS and Snowflake. Product services emit events through EventBridge and Kinesis Firehose into S3, auto‑ingested into Snowflake via Snowpipe, while batch and third‑party data lands through Snowpark‑deployed Python procedures. All infrastructure is defined in Terraform/Terragrunt, pipelines are built in Python, CI/CD runs on GitHub Actions, and Datadog provides end‑to‑end pipeline observability.

The Role

• Own and evolve the event ingestion path — EventBridge to Firehose to S3 to Snowpipe — including partitioning and throughput tuning as event volumes grow across streams such as fraud, payments and merchant.


• Design and deploy Snowpark‑based Python procedures for file‑based and third‑party data loads, scheduled and monitored as Snowflake Tasks.


• Work with product engineers to manage upstream schema changes and late‑arriving data without breaking downstream consumers.


• Own and extend Terraform/Terragrunt modules for ingestion infrastructure across multiple environments.


• Design Datadog monitors and alerting for Firehose, EventBridge, DLQs and Snowpipe, and triage ingestion failures when they occur.


• Manage PII handling, masking, data governance and access in Snowflake, supporting audit‑ready reconciled data for Finance and Compliance.


• Build privacy‑first data pipelines using dbt and Snowflake Dynamic Tables, with testing and monitoring that catches issues before they reach the business or the Analytics Engineering team.

What We’re Looking For

• Hands‑on experience with streaming and event ingestion on AWS — Kinesis, EventBridge or equivalents — plus S3 partitioning and queue/DLQ patterns.


• Terraform experience is essential; owning modules across environments rather than just consuming them. Terragrunt is a plus.


• Strong SQL on Snowflake, including Snowpipe, Tasks and Snowpark.


• Proven dbt experience building well‑tested SQL and Python models.


• Experience with GitHub Actions (or an equivalent) and with secrets management, PII and sensitive data handling in Snowflake.


• Python skills for pipeline and infrastructure code — typed models, testing and CI‑deployed services rather than just scripting.

You’ll need a pragmatic, systems‑level mindset and the ability to explain pipeline failures, infrastructure trade‑offs and data‑flow issues clearly to both engineers and non‑technical stakeholders. This role suits someone who takes full ownership of what they build, from design through to production reliability.

This could suit a data engineer at a scale‑up or a data‑heavy fintech who has outgrown ticket‑driven work and wants to own an entire ingestion platform end to end.

#Referment