Data Engineer

Mirus; Greater London; On-site; GBP 90,000 - 130,000; Full time

Job summary

Mirus is seeking a data engineering professional to own the end-to-end production pipelines, from source capture to delivery to clients. You will bring all processes under GitHub, implement automated testing and CI/CD, and adapt the platform as needed to meet changing requirements.

The role involves working with Azure Databricks, Snowflake, SQL Server and Databricks, and shaping the architecture for ingestion, processing and delivery to support scalable, secure and efficient data workflows.

Qualifications

• Experience building production data pipelines end-to-end.
• Strong Python and SQL for data engineering tasks.
• Hands-on change data capture and event processing.
• Experience with Azure Databricks, Databricks Lakehouse.
• GitHub and CI/CD for data workloads.

Responsibilities

• Own end-to-end production pipelines: source capture, transformation and delivery.
• Define architecture for ingestion, processing and delivery to clients.
• Build batch and streaming pipelines from databases, APIs, and event streams.
• Implement change data capture, incremental loads and schema changes.
• Maintain data contracts, reconciliation and production readiness.

Skills

Python programming

SQL

Data pipelines

CI/CD

GitHub

Data quality

Incident response

Multi-tenant data platforms

Data modelling

Tools

Snowflake

Databricks

Azure Databricks

ADLS

GitHub Actions

Kafka

Snowflake CLI

dbt

Job description

I'm working with a specialist data and technology business in the private markets space. Their platform sits behind the valuation and monitoring work their clients rely on, so the data has to be right, on time, every time.

The role

You'll own the production pipelines end to end: source capture, transformation, reconciliation and delivery to clients. A big part of the job is bringing all of it properly under GitHub, automated testing and CI/CD.

Today the platform captures change data from relational systems, processes it on Azure Databricks, and delivers into Snowflake, SQL Server and Databricks. The capture method isn't set in stone and may change.

What you'll be doing

• Helping redefine the architecture across ingestion, processing and delivery, so it's stable, secure and quick enough for more advanced use cases
• Building and running batch and streaming pipelines from databases, APIs, event streams and semi-structured sources
• Implementing change data capture and incremental loads, handling ordering, deletes, replay and slowly changing dimensions
• Building medallion datasets and dimensional models, delivered into Snowflake, SQL Server and Databricks
• Putting data contracts, reconciliation and row-level quarantine in place before anything is published
• Owning the GitHub workflow and CI/CD: tests, reviews, environment promotion and deployment as code
• Getting to the bottom of production data failures when they happen
• Turning requirements from product, valuation and client-facing teams into pipelines that actually run well in production

What you'll bring

• Snowflake or Databricks as your main platform, including modelling, performance tuning and keeping costs under control
• Strong Python and SQL for building and testing pipelines
• Hands-on change data capture and event processing, including ordering, replay and schema change
• Azure experience, including Databricks, ADLS and private network connectivity
• GitHub and CI/CD for data workloads, with GitHub Actions or similar
• A solid grip on data quality, reconciliation, monitoring and production incident response
• Experience building secure, multi-tenant data platforms, including tenant isolation, access control and data protection

Great if you also have

• Databricks Lakeflow, Auto CDC, Declarative Automation Bundles and DQX, or the Snowflake equivalents (Dynamic Tables, Streams and Tasks, Snowpark, Snowflake CLI deployments, Data Metric Functions)
• Debezium, Kafka Connect or Confluent Kafka. This is the current ingestion path, though it may be replaced
• Apache Airflow or a similar orchestrator
• Kafka or Spark Structured Streaming, Apache Iceberg or Delta Sharing, and dbt for analytical models on curated data
• Private markets data: valuations, funds, portfolio companies or capital activity
• Confidence working directly with client technical teams, and collaborating across engineering, product and other stakeholders