Senior Data Engineer — Databricks, Snowflake & RegTech Impact

Silversmith Capital Partners; Greater London; On-site; GBP 90,000 - 130,000; Full time

Benefits offered by this job

Private Medical Insurance (via AXA)

Life Insurance - 4× salary

Flexible hours

Paid volunteering leave

Gym & fitness discounts

Office snacks and drinks

Job summary

Novatus, a RegTech SaaS and consulting practice, seeks a Senior Data Engineer to advance our configuration-driven data platform in Databricks and Snowflake for regulated use cases. You will write clean, well-tested code and provide technical leadership in a growing team.

You will help modernize our data infrastructure as we scale, migrating from Python scripts to Databricks and Snowflake, and ensure auditability, data quality, and reliability across client deployments.

Qualifications

• 8+ years as a software engineer, including 5+ years in data engineering.
• Experience designing auditable, reproducible data pipelines in regulated or high-integrity environments using Python and PySpark.
• Able to write and optimize complex SQL queries on large data sets.
• Strong data modeling and warehouse design fundamentals.
• Strong software engineering fundamentals: clean code, DRY, SOLID, automated testing, CI/CD, and observability.
• Deep experience with a cloud lakehouse / data warehouse (Databricks, Snowflake, BigQuery, Redshift, EMR).
• Experience with a workflow orchestrator (Airflow, Dagster, or Databricks Workflows).
• Proven system and platform design judgment with ability to own architectural decisions.

Responsibilities

• Design and evolve data framework in Databricks and Snowflake.
• Coach and mentor junior engineers, and lead design reviews.
• Design, build, and optimize data pipelines using Databricks, Python and PySpark, with Kafka.
• Ensure pipelines are auditable, lineage-aware, idempotent, and resilient in regulated environments.
• Implement robust data quality controls including testing, validation, monitoring, and alerting.
• Drive performance optimization across Spark and Snowflake workloads; collaborate with Product, Engineering, DevOps, and Regulatory teams.
• Contribute to the continuous improvement of our development processes and tools.

Skills

Databricks

PySpark

SQL

Data modeling

CI/CD

Testing & observability

Python

Cloud data lakehouse

Kafka

Leadership

Tools

Airflow

Snowflake

Databricks

Terraform

BigQuery/Redshift

Job description

Novatus, a RegTech SaaS and consulting practice, seeks a Senior Data Engineer to advance our configuration-driven data platform in Databricks and Snowflake for regulated use cases. You will write clean, well-tested code and provide technical leadership in a growing team.

You will help modernize our data infrastructure as we scale, migrating from Python scripts to Databricks and Snowflake, and ensure auditability, data quality, and reliability across client deployments.